Tactile, visual, and array controllers for real-time control of music signal processing, mixing, video, and lighting

ABSTRACT

A system and method for real-time controlling of signal processors, synthesizers, musical instruments, MIDI processors, lighting, video, and special effects in performance, recording, and composition environments using images derived from tactile sensors, pressure sensor arrays, optical transducer arrays, chemical sensor arrays, body sensor arrays, and numerical computation. The invention provides for null/contact touch-pads, pressure sensor arrays, and body sensor arrays as tactile control interfaces, for video cameras and light sensor arrays as optical transducers, for chemical sensor arrays, and for other numerical image generation from computer processing or numeric simulation. Tactile transducers may be put on instrument keys of conventional instruments, be attached to existing instruments, or be used to create entirely new instrument or controller configurations. Chemical sensor arrays and for other numerical image generation from computer processing or numeric simulation can be used to monitor or simulate natural physical phenomenon such as self-organizing process behavior or environmental conditions. Arrays of scalar or vector values are processed to extract pattern boundaries, geometric properties of pixels within the pattern boundaries (geometric center, weighted moments, etc.), and higher-level derived information (senses of rotation, segmented regions, pattern classification, syntaxt, grammers, sequences, etc.) which are used to create control signals to external audio, visual, and control equipment or algorithms. The invention also provides for MIDI and non-MIDI control signals.

RELATED CASES

[0001] This application is a divisional continuation of U.S. applicationSer. No. 09/313,533, filed May 15, 1999, in turn based on U.S.Provisional Serial No. 60/085,713 filed May 15, 1998.

BACKGROUND OF THE INVENTION

[0002] 1. Field of Invention

[0003] This invention relates to musical instrument performance systemsand environments, and in particular to the combination of novelinstrument entities built from synergistic arrangements of traditionaland novel instrument elements, and the interconnection of saidinstrument entities utilizing generalized interface entities to signalrouting, processing, and synthesis entities built from synergisticcombinations of traditional and novel architectures, processes, andmethodologies.

[0004] The systems and methods herein are intended to make possible anew generation of musical instrument products with enhanced capabilitiesand sounds, new semiotic-oriented performance capabilities, and richcomposition and recording environments.

[0005] 2. Background

[0006] There has been considerable advancement in music technology inthe last several decades, but recent innovations driven by mass-marketforces have narrowed the range of possibilities for commerciallyavailable instruments and the ways in which new recorded and performedmusic are being explored. Audio samples of diverse instruments, advancedsignal processing power, improved fidelity, the MIDI control interface,sequencers, and music workstations are important assets but, togetherwith the ways synthesizers, signal processing systems, and instrumentcontrollers have come to be designed, the channel of innovation isfocused on a relatively narrow conceptual range that will consume asmuch rework and refinement energy as can be allotted. A few modernoutlier innovations have appeared, such as the Roland COSM signalprocessing methods, Yahama VL1 model-based synthesis methods, andBuchla's and Starr Switch alternative MIDI controllers, but due to thefocused drive of the mainstream these exceptions are largely orphaned intheir application.

[0007] What is needed is some reach into the souls (rather than makesamples) of deep non-Western and Western instruments, a recasting of thenow institutionalized signal processing chains, adaptations of newclasses of applicable physical phenomenon, extensions as to the typesand forms of meaningful human control, and, in the context ofperformance, a deeper integration of visual and audio environments.

SUMMARY OF THE INVENTION

[0008] Based on research and development of this nature, it is possibleto create a new-generation framework for expanding the timbral,expressive range, artistic depth, and semiotic aspects of performed andrecorded music as well as wide ranges of performance art. Such aframework is particularly advantageous if it were to build on andinter-work with both the existing music technology mainstream and thelong established playing techniques of expressively sophisticated,iconic, or significantly adaptable instruments. With such attributes,isolated products and musical directions can be gently folded in to theestablished mainstream and evolve as the mainstream finds moments ofstagnation and boredom within itself. This methodology would permit thecurrent manufacturing and marketing establishments of music technologyand content to progressively and profitably shift to a more creativelysatisfying and sustainable path.

[0009] To these ends, the invention provides methods, apparatus, andexample implementations subscribing to a standardized framework whichaddress these needs and opportunities.

[0010] 1. A key aspect of the invention is a unified architectureinvolving instrument entities, generalized instrument interfaces, andsignal routing, processing, and synthesis elements.

[0011] 2. A further aspect of the invention is the defining of generalinstrument elements which instrument entities can be created from.

[0012] 3. A further aspect of the invention is augmenting existinginstruments lending themselves to expansion with said general instrumentelements.

[0013] 4. A further aspect of the invention is the use of miniaturekeyboards for the attachment to existing instruments.

[0014] 5. A further aspect of the invention is the expansion ofkeyboards to include any one or more of proximate, superimposed,programmable tactical feedback, and/or multiple (more than 2) parameterkey features.

[0015] 6. A further aspect of the invention is the sharing of sameelectronics across multiple keyboards and/or strum-pads.

[0016] 7. A further aspect of the invention is that of strum-pads withnon-repeating contacts along the strum path and flexible assignment ofnote event control signals to each contact.

[0017] 8. A further aspect of the invention is that of includingstandardized arrangements of panel controls, such as switches andsliders, to instruments.

[0018] 9. A further aspect of the invention is the use of null/contacttouch-pads, potentially fitted with impact and/or pressure sensors andwith the potential derivation of multiple contact point information, asa musical interface.

[0019] 10. A further aspect of the invention is that of pressure-sensorarray touch-pads as an instrument controller, potentially includingimage recognition capabilities and the ability to derive and assigncontrol parameters from the way the pad is contacted

[0020] 11. A further aspect of the invention is the structuring ofassociated image processing for a pressure-sensor array touch-pad tocapture hand and foot contact postures and gestures

[0021] 12. A further aspect of the invention is the structuring ofassociated image processing for a pressure-sensor array touch-pad toderive parameters from hand and foot contact postures which permit theapplication of useful metaphors in their operation.

[0022] 13. A further aspect of the invention is the implementation ofpressure-sensor array touch-pads, and potentially related decentralizedimage processing and networking functions, in a mini-array chip whichcan be tiled into arbitrary shapes, potentially including instrumentkeys.

[0023] 14. A further aspect of the invention is using key displacementtogether with contact position to derive at least three parameters froma standard Western keyboard key.

[0024] 15. A further aspect of the invention is a foot controller withbuttons and pedals that have associated alphanumeric displays.

[0025] 16. A further aspect of the invention is a foot controller withany one or more of: hierarchical organization of changeable storedprogram elements, arbitrary button assignment of hierarchy controlfunctions, and/or multiple interpretation geometric layout of buttonsand pedals.

[0026] 17. A further aspect of the invention is a method for doing onehanded drum rolls with acoustic drums or multiple parameter electronicdrumpads.

[0027] 18. A further aspect of the invention is: mallets, beaters, andbows with any one or more of impact, grip, position, or pressure,strain, and/or motion sensors.

[0028] 19. A further aspect of the invention is an autoharp adaptationwith both strings and strumpads.

[0029] 20. A further aspect of the invention is: a string autoharpadaptation where chord buttons issue control signals.

[0030] 21. A further aspect of the invention is an autoharp adaptationwhere a note-oriented keyboard is used to replay multiple note chordbuttons, potentially where the keys are multiple parameter keys.

[0031] 22. A further aspect of the invention is: autoharp, Pipa, Koto,Harp, Mbira, pedal steel, and Sitar adaptations with separate pickupsfor each vibrating element, potentially also employing pitch shifting onselected vibrating element.

[0032] 23. A further aspect of the invention is: Pipa, Koto, Harp,Mbira, pedal steel, and Sitar adaptations with strum-pads.

[0033] 24. A further aspect of the invention is: guitar, Pipa, Koto,Harp, Mbira, pedal steel, and Sitar adaptations with vibrating elementexcitation drivers built into the instrument.

[0034] 25. A further aspect of the invention is: guitar, Pipa, Koto,Harp, Mbira, pedal steel, and Sitar adaptations with additional stringarrays and/or one or more miniature keyboards with keys close to thestring array.

[0035] 26. A further aspect of the invention is the use of vowelsynthesis in conjunction with a bowed instrument.

[0036] 27. A further aspect of the invention is attaching a video camerato an instrument.

[0037] 28. A further aspect of the invention is the use of opticalpickups for metalaphones and drum heads.

[0038] 29. A further aspect of the invention is the use ofnon-equilibrium chemical reactions as musical controllers or parts ofinstruments.

[0039] 30. A further aspect of the invention is the use of photoacousticphenomena as musical controllers or parts of instruments.

[0040] 31. A further aspect of the invention is the use of video camerasas musical controllers and/or instruments.

[0041] 32. A further aspect of the invention is a wide variety of newsignal processing innovations, including spatial timbre construction,hysteretic waveshaping, layered signal processing, location modulationof signal pan constellations, cross-product octave chains.

[0042] 33. A further aspect of the invention is the provision for a widevariety of control signal monoatic and polyadic operations as listed inthe disclosure.

[0043] 34. A further aspect of the invention is the provision for a widevariety of control routing capabilities as listed in the disclosure,including routing at MIDI message index levels.

[0044] The system and method herein can be applied to live performance(music, dance, theater, performance works, etc.), recorded audio andvideo production, and composition.

[0045] The invention will be described in greater detail below withreference to the accompanying drawings.

DESCRIPTION OF THE DRAWINGS

[0046]FIG. 1 shows a general overview of the invention.

[0047]FIG. 2 shows examples of internal interconnections among thefunctional grouping elements within an instance of a signal routing,processing, and synthesis entity, all shown in FIG. 1.

[0048]FIG. 3 shows an example of a proximate keyboard array.

[0049]FIG. 4 shows an example of an instrument-mounted miniaturekeyboard configuration employing one miniature keyboard, in particularan adapted Indian sitar with many additional example instrumentelements.

[0050]FIG. 5 shows an example of an instrument-mounted miniaturekeyboard configuration employing two miniature keyboards, in particularan adapted electric guitar with many additional example instrumentelements.

[0051]FIG. 6 illustrates an arrangement where a dedicated continuous ornear-continuous sensor is attached to each key so as to instantaneouslymeasure the displacement of the attached key.

[0052]FIG. 7 illustrates an arrangement by which programmable tactilefeedback can be applied to a key, either in conjunction or without acontinuous or near-continuous sensor to measure key displacement.

[0053]FIG. 8 illustrates a shared scanning arrangement supporting aplurality of any of keyboards, strum-pad, buttons, switches, etc.

[0054]FIG. 9 illustrates an example method for realizing a flexiblegeneralized strum-pad element and associated stored program control.

[0055]FIG. 10 shows an example implementation of both generalized andspecific control signals derived from panel controls, actuators, andsensors using MIDI.

[0056]FIG. 11 shows an example of how two independent contact points canbe independently discerned, or the dimensional-width of a single contactpoint can be discerned, for a resistance null/contact controller with asingle conductive contact plate or wire and one or more resistiveelements whose resistance per unit length is a fixed constant througheach resistive element.

[0057]FIG. 12 shows an example implementation of both generalized andspecific control signals derived from electrical contact touch-padsemploying MIDI messages as the output control signal format.

[0058]FIG. 13 shows how a pressure-sensor array touch-pad can becombined with image processing to assign parameterized interpretationsto measured pressure gradients and output those parameters as controlsignals.

[0059]FIG. 14 illustrates the positioning and networking of pressuresensing and processing “mini-array” chips in both larger contiguousstructures and in isolated use on instrument keys, instrumentfingerboards, and instrument bodies.

[0060]FIG. 15 illustrates the pressure profiles for a number of examplehand contacts with a pressure-sensor array.

[0061]FIG. 16 illustrates how six degrees of freedom can be recoveredfrom the contact of a single finger.

[0062]FIG. 17 illustrates the regions of vowel sounds associated withparticular resonant frequency combinations in vowel sound production.

[0063]FIG. 18 illustrates an example two-dimensional timbre space intraditional instrument orchestration.

[0064]FIG. 19 shows an example of keys from a traditional Westernkeyboard fitted with multiple uniformly-sized pressure-sensing andprocessing “mini-array” chips.

[0065]FIG. 20 shows electromagnetic, Hall-effect, piezo, and opticalpickup methods for deriving separate audio signals for each vibratingelement of a multiple vibrating element instrument entity.

[0066]FIG. 21 shows how an off-bridge buzz-plate can be combined with apiezo bridge sensor in replacement of a gradient buzz-bridge so as topermit the use of non-ferromagnetic strings.

[0067]FIG. 22 shows the basic idea of controlled feedback as used inrecent contemporary music.

[0068]FIG. 23 shows an example implementation of a simple approach forreplacing acoustic excitation of a vibrating element withelectromagnetic excitation.

[0069]FIG. 24 shows various combinations of piezo and electromagneticvibrating element pickups and exciters for separately controllableexcitation of each vibrating element.

[0070]FIG. 25 shows adding signal processing for spectral and amplitudecontrol of electromagnetic excitation.

[0071]FIG. 26 shows multiple vibrational elements with commonelectromagnetic excitation.

[0072]FIG. 27 illustrates examples of single, double, and quadrupletouch-pad instruments with pads of various sizes and supplementalinstrument elements.

[0073]FIG. 28 illustrates some enhanced foot-pedal arrangements whichpermit simultaneous single-foot adjustment of a plurality of continuousrange parameters for use with floor controllers.

[0074]FIG. 29 shows some example layouts involving 2 geometric regionsfor a moderate number of foot operated controllers and 4 geometricregions for a larger number of foot operated controllers.

[0075]FIG. 30 shows an example large-scale arrangement of two impactsensors and/or touch pads for executing one-handed drum-rolls andderiving large amounts of control information.

[0076]FIG. 31 shows an example of an enhanced autoharp implementation asprovided for in the invention.

[0077]FIG. 32 shows how the autoharp arrangement of FIG. 31 can beadjusted to replace its chord button array and associated strum-padswith a keyboard and one or more strum-pads positioned over the keyboard.

[0078]FIG. 33 shows an example Koto implementation provided for inaccordance with the invention.

[0079]FIG. 34 shows an example Mbira implementation provided for inaccordance with the invention.

[0080]FIG. 35 shows an example electric guitar implementation inaccordance with the invention based on a Gibson model ES-335 guitar; theinvention's enhancements shown can be added on as modules, addedcollectively, or built-in.

[0081]FIG. 36 shows an example of an adapted European arch-lute with amix of single strings and double string pairs.

[0082]FIG. 37 shows an example pedal steel guitar adaptation as providedfor by the invention.

[0083]FIG. 38 shows an example flat-necked instrument with a five stringsection emulating a sitar string arrangement and several additionalstrings used for bass or other accompaniment.

[0084]FIG. 39 shows an example multiple-pitch sympathetic/buzz/twangresonator using banks of short audio delays with high resonances tunedto each selected pitch, each followed by a dedicated low-speed sweepingflanger with moderate resonance, a dedicated low-speed sweeping flangerwith moderate resonance, and a low-speed auto-panner.

[0085]FIG. 40 shows an example adapted Chinese Pipa as provided for bythe invention featuring a keyboard, strum-pad, touch-pad, slider array,switch array, and impact sensors.

[0086]FIG. 41 shows another example adapted Chinese Pipa as provided forby the invention featuring a bass string array, a harp string array, andimpact sensors.

[0087]FIG. 42 shows a bow fitted with sensors to gather information fromthe hand, bow hairs, and bow motion.

[0088]FIG. 43 shows adaptations of a flute and recorder layout withpressure sensors replacing key sites, air turbulence measurements, andair pressure average measurements as provided for in the invention.

[0089]FIG. 44 shows how an optical pickup may be created for a suspendedgong; this technique may also be used for many other types ofmetallophones.

[0090]FIG. 45 shows example gong arrays as part of a one-hand ortwo-hand percussion instrument stand.

[0091]FIG. 46 illustrates spatial arrays of electrodes which may be usedfor measurement, as well as control, in two-dimensional andthree-dimensional configurations.

[0092]FIG. 47 shows an arrangement where evolving chemical patterns inthe dish of FIG. 46 are illuminated with light sources and visuallymonitored by an overhead camera for any one or more of controlextraction, visual display, or visual recording.

[0093]FIG. 48 illustrates example optical measurements of photoacousticphenomena in applicable materials which may be converted to electricalsignals and an example electro-acoustic measurement of photo-inducedacoustic phenomena in applicable materials.

[0094]FIG. 49 shows how generalized interfaces can be built in whole orvia separable parts which may be used selectively as needed orappropriate.

[0095]FIG. 50 shows multiple vibrational elements with multi-channeltransducers applied directly to stereo or multi-channel mix-down.

[0096]FIG. 51 shows multiple vibrational elements with multi-channeltransducers and individual signal processing prior to mixing.

[0097]FIG. 52 shows addition of a control signal extraction element tothe arrangement of FIG. 51.

[0098]FIG. 53 shows partial mix-downs of vibrating element signals fedto a number of signal processors and straight-through paths in route tosubsequent mix-down.

[0099]FIG. 54 shows a switch matrix assigning signals from vibratingelements to a number of signal processors en route to subsequentmix-down.

[0100]FIG. 55 shows a more flexible method for providing signalprocessors with vibrating element signals and other signal processoroutputs via switch matrix, and additional partial mix-downs by replacingsaid switch matrix with a mixer.

[0101]FIG. 56 shows configuration control of signal processors, mixers,and switch matrix, and synthesizer interfaces via logic circuitry and/ormicroprocessing.

[0102]FIG. 57 shows a very general combined environment formulti-channel signal processing, mixing, excitation, and program controlof overall configuration.

[0103]FIG. 58 shows a stereo-input, stereo output configuration of twomonaural flange and/or chorus elements wherein the unaltered signal ofeach input channel is combined with a delay-modulated signal from theopposite channel.

[0104]FIG. 59 illustrates a combination of a spatialized effect, twodistortion elements, and a stereo (N=M=2) cross-channel modulated delay.

[0105]FIG. 60 illustrates examples of inhomogeneous layered signalprocessing which may be used as shown, with selected omissions, or as anarchtype for similar constructions.

[0106]FIG. 61 illustrates an example of a generalized hysterisis modelconstruction as provided for by the invention.

[0107]FIG. 62 shows an example implementation of a cross-product octavechain particularly suited to low cost implementation with logic chips orsimple DSP program loops.

[0108]FIG. 63 illustrates an example flexible control and configurationhierarchy for control signal and stored program handling andorganization.

[0109]FIG. 64 shows an example method for the generation of controlsignals from fundamental and overtone information in a signal from avibrating element of fixed known pitch.

[0110]FIG. 65 shows combining and/or processing fundamental and overtoneinformation obtained from a vibrating element signal prior to parameterextraction.

[0111]FIG. 66 shows an example implementation of an adaptive method fortracking overtones for a variable-pitch vibrating element with knownovertone series.

[0112]FIG. 67 illustrates an example approach wherein a plurality ofLFOs with features as prescribed by the invention may be implemented.

[0113]FIG. 68 illustrates traditional stage lighting elements includingover-heads, far-throw, foot, back, floor.

[0114]FIG. 69 illustrates example instrument lighting.

[0115]FIG. 70 illustrates example rotating speaker emulation lightsculptures.

[0116]FIG. 71 illustrates light pyramid arrays and light columns arrays.

[0117]FIG. 72 illustrate stage video projection arrangements.

DETAILED DESCRIPTION

[0118] 1 Overview

[0119] The invention relates to a collection of instruments (adapted,electronic, or combined), generalized instrument electrical interfaces,control signal extraction and manipulation systems, musical synthesismodules, layered audio signal processing, lighting control, lightsculptures, instrument lighting effects, video control, and videodisplay. The resulting rich sonic and visual environment can be used forlive performance, recorded audio and video production, and composition.

[0120]FIG. 1 shows a general overview of the invention which, at itshighest level, consists of one or more instances of instrument entities100, generalized interface entities 110, and signal routing, processing,and synthesis entities 120. It is understood that the invention providesfor the possibility of several instances of each of these entities. Forexample, several instruments 100 may be supported (adapted guitar,adapted sitar, adapted autoharp, touchpad/slider/switch controller,etc.) by each 110/120 system; further, for each collection of pluralityof instruments 100 and signal routing, processing, and synthesisentities 120, there may be both full-feature interface cables orsimplified reduced-feature cables implementing versions of thegeneralized interface 110; finally, for each collection of instruments100 and generalized interface entities 110 there may be various versionsof signal routing, processing, and synthesis entities 120 (smallperformance systems, large performance systems, studiorecording-oriented or composition-oriented systems, etc.).

[0121] 1.1 Instrument Overview

[0122] In more detail, each instrument entity 100 in general internallyconsists of one or more elements. The elements fall into two broadcategories, namely those that produce audio-frequency signals and thosethat instead produce only control signals. Of these, it is also possibleto derive control signals from the audio-frequency signals (reflectingpitch, amplitude, relative harmonic content, etc.). Control signals,regardless of their origin, in general are used to control theprocessing, replay, or synthesis of audio-frequency signals; however,the control signals can also be used to control lighting, video, specialeffects, etc.

[0123] Referring to FIG. 1, the instrument 100 may contain internalpower sources (such as batteries, large-value capacitors, etc.) and/orpower regulation elements 101. Next the control signal sources that maybe included within an instrument entity 100 may be of a traditionaltechnology or nature, such as knobs, keys, switches, touch-pads,sliders, buttons, sensors, etc.; these will be termed electronicinterface elements 102. In addition, it is also possible to generatecontrol signals from more exotic processes such as chemical oscillators,chemical chaos, photoacoustic, environmental sensors, etc. These will betermed alternative control signal elements 103. The audio-frequencysignal sources that may be included within an instrument also broadlyfall into two classes. One class is that of traditional vibratingelements (strings, tynes, surfaces, solid volumes, air columns, etc.)whose mechanical audiofrequency vibrations can be electrically sensedvia electromagnetic, photo-electric, piezo, Hall-effect, or other typesof sensors or transducers. In many cases it is possible to excite thesemechanically vibrating elements by electronic methods (magnetic fields,piezo transducers, etc.) or electronically controlled methods (motorizedbowing, solenoid strikers, etc.) This first class of audio-frequencysignals sources will be termed sensed/excited vibrating elements 104. Inaddition, it is also possible to generate audio-frequency signals frommore exotic processes such as chemical oscillators, chemical chaos,photoacoustic, environmental sensors, etc. These will be termedalternative control signal elements 105.

[0124] Finally, the instrument may also contain various additionalvideo, lighting, and special effect elements 106.

[0125] 1.2 Generalized Interface Overview

[0126] Again referring to FIG. 1, the invention provides for bothinstrument entities 100 and signal routing, processing, and synthesisentities 120 to be fitted with compatible electrical interfaces, termedgeneralized instrument interfaces or (or more concisely, generalizedinterfaces) 110, which can exchange any of the following:

[0127] incoming electrical power (111)

[0128] outgoing control signals from switches, controls, keyboards,sensors, etc., typically in the form of MIDI messages but which may alsoinvolve contact closure or other formats (112)

[0129] control signals to lights, pyrotechnics, or other special effectelements within and/or attached to the instruments, said signals beingeither in the form of MIDI messages, contact closure, or other formats(113)

[0130] outgoing audio signals from individual audio-frequency elementsor groups of audio-frequency elements within the instruments (114)

[0131] incoming excitation signals directed to individualaudio-frequency elements or groups of audio-frequency elements withinthe instruments (115)

[0132] outgoing video signals (such as NTSC, PAL, SECAM) or imagesignals sent from the instrument (116)

[0133] incoming video signals (such as NTSC, PAL, SECAM) or imagesignals sent to the instrument for purposes such as display or as partof a visually controlled instrument (117).

[0134] The interfaces may be realized by one or more of any ofconnectors, cables, fibers, radio links, wireless optical links, etc.

[0135] 1.3 Signal Routing, Processing, and Synthesis Overview

[0136] Referring to FIG. 1, the invention provides for one or moresignal routing, processing, and synthesis entities 120. These entitiesfirst route and process received audio-frequency, control, and videosignals. Additionally, these entities 120 may extract control signalsfrom received audio-frequency and video signals, perhaps under thedirection of selected control signals. Finally, these entities 120 mayalso synthesize audio-frequency, control, and video signals, typicallyunder the direction of selected control signals.

[0137] Again referring to FIG. 1, the signal routing, processing, andsynthesis entities 120 internally may include:

[0138] power supplies 121 for both internal and instrument powering

[0139] control signal routing 122 for interconnecting control signalsources with control signal destinations

[0140] control signal processing 123 for instantaneous control messagetransformations (such as inversions) and inter-operations (such asaveraging, adding, multiplication, etc.)

[0141] audio signal routing 124 for interconnecting audio signal sourceswith audio signal destinations

[0142] audio signal processing 125 for (typically real-time)transformations, typically under real-time control via selected controlsignals

[0143] video signal routing 126 for interconnecting video signal sourceswith audio signal destinations, typically under real-time control viaselected control signals

[0144] video signal processing 127 for (typically real-time) videosignal transformations, potentially under real-time control via selectedcontrol signals

[0145] control signal extraction 128 a for the derivation of (typicallyreal-time) control signals from audio or video signals, potentiallyunder real-time control via selected control signals

[0146] control signal synthesis 128 b for the internal creation oftime-varying control signals (such as low-frequency control oscillators,envelop generators, slew limiters, etc.), potentially under real-timecontrol via selected control signals

[0147] audio signal synthesis 129 a, typically under the direction ofselected control signals, and typically as per conventional musicsynthesizer hardware and software

[0148] video signal synthesis 129 b, typically under the direction ofselected control signals.

[0149] program storage 130 for storing configuration programs and eventsequences

[0150] In FIG. 1 it is understood that the elements 121 through 130represent functional groupings and not necessarily hardware-centralizedor software-centralized subsystems.

[0151]FIG. 2 shows examples of internal interconnections among thefunctional grouping elements 121 through 130 within an instance of asignal routing, processing, and synthesis entity 120. In FIG. 2, asbefore, it is understood that the elements 121 through 130 representfunctional groupings and not necessarily hardware-centralized orsoftware-centralized subsystems.

[0152] In the example interconnections, power is distributed throughoutvia functional fan-outs 131; here it is understood that there many bemany decentralized power supplies for the individual subsystemscomprising or implementing elements 122-130. Program store informationis also distributed throughout via paths 132 (associated with specificsubsystems of elements 122-129) and/or path 133 to the control signalrouting element 122; typically both methods are used as portions of theprogram control may be stored within individual elements 122-129 andportions may reside within one or more centralized program storesubsystems (such as MidiTemp model MP-88, Digital Music Corporationmodel MX-8, controlling PC, etc.), comprising 130.

[0153] 1.4 Remaining Document Overview

[0154] With this overview complete, the remainder of the discussion isorganized as follows. The next four Sections concern instruments 100.First, a number of instrument element and instrument subsystems aredescribed. Two subsequent sections then describe a large number ofexample instruments that are perfected through applicable combinationsand arrangements of the aforementioned instrument elements andsubsystems of elements; the first of these sections purely electroniccontrollers while the second addresses adaptations of conventionalinstruments with special attention paid to specific nuances andopportunities within those instruments. Following this, some alternativeaudio and control signal sources are then considered.

[0155] Next the general instrument interface 110 is then considered inadditional detail. A subsequent section then addresses the signalprocessing, and synthesis entities 120. A final section provides a fewexample envisioned applications of the invention.

[0156] 2 Instrument Elements and Instrument Subsystems

[0157] The invention includes a number of electronically interfacedinstruments used by one or more performers.

[0158] These instruments involve either pure electronic interfacesarranged to form an instrument, vibrating elements which typically arein arrangements adapted from existing instruments, exoticelectrically-monitored oscillatory elements (such as chemicaloscillators), electronic or numerical chaotic models used as sources, orcombinations of these laid out in an artistically operative andergonomic fashion. Vibrating elements within an instrument may also bemade to vibrate via electronically controlled or induced excitation frommagnetic field, piezo electromechanical, or other electronically-drivenor electronically-controlled excitation.

[0159] In general an instrument consists of one or more instrumentelements which may be of one more differing types or classes. Theseinstrument elements may be thought of as subsystems within theinstrument. For example, a 6-string guitar has six vibrating strings;each string is an example of a vibrating element. A singleelectromagnetic or piezo pickup may be used to amplify the entire groupof six strings. The guitar may also have separate electromagnetic orpiezo pickups for each string, as is commonly done for adding a MIDIinterface to an existing electric guitar. This example guitar thensimultaneously has six vibrating elements, one group-pickup subsystem,and six single-string pickup subsystems.

[0160] The guitar may be further enhanced with MIDI-command issuingcontrols, such as knobs, switches, joysticks, touch-pads,motion/position sensors, etc.; these represent an additional subsystem.A reduced-size musical keyboard may be added to the guitar, representingyet another subsystem.

[0161] Specific classes of instrument elements and/or instrumentsubsystems are described in the subsections that follow.

[0162] 2.1 Electronic Interface Instrument Elements and Subsystems

[0163] This class of instrument elements and instrument subsystems donot create audio frequency phenomenon directly but are rather used tocontrol one or more music synthesizers, audio mixers, and/or signalprocessing functions.

[0164] 2.1.1 Proximate, Miniature, and Superimposed Keyboards

[0165] Standard western keyboards found on pianos, harpsichords, organs,and synthesizers are widely used as a human interface for electronicmusical instruments. Some instruments, such as organs and harpsichords,have traditionally (for centuries) included two or more such keyboardsto allow the instrument player to rapidly select among two or moretimbres or ranges. The spacing of the keyboards is almost withoutexception found to be far enough apart that a hand must be committeduniquely to a given keyboard for the moments that the keys are played.This is due to the fact that the bulk of apparatus under the keyboards,keyboard frame, etc. prevented the keyboards from being mounted veryclose together, re-enforced by the fact that music has been composed forplaying at most one keyboard with a given hand (although in virtuosopieces a given hand may very rapidly jump among keyboards). One aspectof the invention expands the usage of traditional keyboards by removingthis limitation via various means.

[0166] 2.1.1.1 Proximate Keyboard Arrays

[0167] One method of implementation is to mount a plurality of keyboardsclose enough together that one hand can, to degrees determined bymechanical details, simultaneously play notes on two or more traditionalkeyboards. There are three methods for increasing the workable proximityof groups of keyboards:

[0168] reduce the vertical separation of the keyboards

[0169] overhang the ends of the white keys on a higher keyboard over thebacks of the white and black keys of a lower keyboard

[0170] reduce the physical length of the keys

[0171] Many modern electronic keyboards have very shallow mechanisms andframes. It is therefore quite straightforward to mount two or morecommonly available electronic keyboards employing either or both of thefirst two methods. With some overhang and (vertically or horizontally)shallow enough mechanisms, it becomes possible to play notes on bothkeyboards simultaneously. In nominal configurations the thumb-to-pinkyreach is nearly the same across both keyboards. Clearly some fingerconfigurations are difficult or impossible across the two keyboards, butthere are also limitations in conventional keyboards that areincorporated in the development of established fingering technique andrespected in keyboard music composition; similar minor techniquedevelopment and compositional respect extensions can be developed forsuch proximate keyboard arrays.

[0172] Without reducing the size of the keyboards a single hand can evenmake invaluable use of three keyboards within a confined range; simpleexample is to add back-up notes of the same pitch or differing octaves.However, two hands may use the two-keyboard playing techniques to makeavid use of a three, four, or more proximate keyboard array.

[0173]FIG. 3 shows an example of a proximate keyboard array. In thisexample, three keyboards 301, 302, 303 are arranged in an overhangingstaircase arrangement. Three views are shown: a side view 300, a topview with hidden key areas suggested by dashed lines 310 a, and the sideview of 300 reoriented as an orthogonal projection 310 b of the top view310 a.

[0174] The separation distances 305 a, 305 b between the tops of thekeys of a given keyboard and the bottoms of the keys of a keyboardoverhanging it should be minimized and in the limit are just slightlylarger than the maximum travel distance of the overhanging key. Thedepth of the overhang 304 a, 304 b is set in the balance between thetrade-off of maximizing desired accessibility to the back of an overhungkey and minimizing the separation distance between the edges of the keysof two adjacent proximate keyboards. It is noted that any of thekeyboards used here may be either of a standard variety or any of themore advanced keyboards described later (miniature, superimposed,multi-parameter keys, pressure-sensor array, etc.). It is also notedthat this technique may be applied to other types of keyboards withapplicable types of key geometry.

[0175] 2.1.1.2 Miniature Keyboards

[0176] If the depth of the keyboard is reduced, the span of a given handis increased further. This may be done by making the keys relativelyshorter, forming a stubby keyboard, or by shrinking the size of theentire keyboard in all dimensions. Such miniature keyboards are commonlyfound on consumer electronic keyboards and keyboard instruments made forchildren.

[0177] Clearly a proximate keyboard array can be created fromminiaturized keyboards. The range of the fingers within and acrossindividual component keyboards may be greatly increased in this fashion,albeit with a perhaps somewhat compromised tradition and technique.

[0178] An additional, and particularly valuable role for the proximatecapabilities of such miniature keyboards is to mount them, as acomponent, on an instrument with other components so as to form a morecomplex instrument where free fingers can operate two or more suchcomponents simultaneously. As a simple example, a guitarist using athumb-pick or classical guitar technique can easily use free fingers toplay chords, bass lines, melodies, etc. on a miniature keyboard attachedto a guitar.

[0179]FIG. 4 shows an example of an instrument-mounted miniaturekeyboard configuration employing one miniature keyboard 421, inparticular an adapted Indian Sitar 400 with many additional exampleinstrument elements which will be described later.

[0180]FIG. 5 shows an example of an instrument-mounted miniaturekeyboard configuration employing two miniature keyboards, in particularan adapted electric guitar 500 with many additional example instrumentelements which will be described later. Here note the two keyboards 521a, 521 b are proximate enough to allow both keyboards and the guitarstrings to be played simultaneously.

[0181] Clearly these methods of miniature keyboard attachment(s) can beapplied to other instruments (Sitar, Pipa, Saz, pedal steel guitar,plucked string bass, etc.) as well as being used to create entirely newtypes of instruments and controllers as will be discussed herein.

[0182] 2.1.1.3 Superimposed Keyboards

[0183] It is also possible to make contact-closure keyboards withmultiple contact sets that actuate at increasing depths of keydepression. Such keyboards may or may not have tactile feedback as toeach level of actuation. Pratt-Read manufactured a “double-touch”keyboard for use in home console organs which closed one set of contactswith a noticeable restoring pressure at about half of the possiblekey-displacement which persisted through full key displacement whereanother contact set closed at the end of key travel. Also, many“velocity sense” keyboards are realized by SPDT switches actuated witheach key; here the beginning of key travel opens a pair of contacts andthe end of key travel closes a second set of contacts, but with nomid-travel tactile feedback.

[0184] In either case, there are one contact closure event at partialkey travel and two events at full key travel. These events can beinterpreted as superimposed keyboards. One example interpretation isthat the first event triggers one synthesizer voice and the second eventriggers a second voice; in this manner keys struck with partialdisplacement sound with only one voice but those struck with fulldisplacement sound both voices. Another example is for a first voice tobe triggered at partial displacement but turned off at fulldisplacement. If the first voice has a long attack, it would be drownedout by the second voice, or in short duration serve as acceptabletransient ornamentation (for example, mimicing a “key click” or “airturbulence chiff”), this arrangement effectively resulting in a partialkey displacement sounding only the first voice and a full keydisplacement sounding only the second voice.

[0185] Note in either arrangement, a fluctuation of the applied keypressure can vary which voices continue to sound (in the firstarrangement, the second voice will go on and off with the first voiceheld; in the second arrangement, the first and second voices willalternate being on or off in a mutually exclusive fashion).

[0186] As the superposition of keyboard principal proves useful in thistwo-level setting, it is natural to consider further extensions of thisapproach to more levels and additional interpretations. In the limit, akeyboard could have a continuous sensor (such as a potentiometer,magnetic or optical gradient, etc.) or near-continuous sensor (such as abinary encoded control) attached to each key. FIG. 6 illustrates anarrangement where a dedicated continuous or near-continuous sensor isattached to each key so as to instantaneously measure the displacementof the attached key. In such an arrangement external electronics woulddefine quantized displacement thresholds to which various superimposedkeyboard interpretations would be assigned.

[0187] As a first bonus, it is also noted that this same continuous ornear-continuous key-displacement sensor arrangement can be used in otheroperational modes to provide other very valuable expressive functions,for example volume or timber control or velocity contour tracking, aswill as will be described in a later section.

[0188] In practice, the two-level superimposed keyboard provides theplayer with tactile feedback as to what point of travel the key hadpassed in the form of a noticeable change in resistive restoringpressure. For a more generalized system as described above andillustrated in FIG. 6, there may be applications where such tactilefeedback is not especially necessary, for example in triggeringadditional synthesizer voices to create an increasing gradient ofrichness as the key is pressed further and further. In othercircumstances, particularly if there are only a few levels implemented,tactile feedback may indeed be desirable, particularly that withdiscernible discrete steps matching the trigger-level quantizationpoints in key travel.

[0189] Highly flexible programmable tactile feedback can be imposedseparately on each key by a dedicated solenoid, motor, pneumatic, fluid,or other means. Less flexible yet still somewhat programmable tactilefeedback could also be had by means of an electrically adjustable globalmechanical arrangement serving all keys in a keyboard, for exampleengaging additional sets of springs or pliable rubber pressure-resistingcones. FIG. 7 illustrates an arrangement by which programmable tactilefeedback can be applied to a key, either in conjunction with or withouta continuous or near-continuous sensor to measure key displacement.Without key position information, an electrically-controlled restoringforce element with built-in levels of key pressure resistance (forexample, by means of a sequence of spatially distributed electromagneticcoils that can be switched on at configuration time to create additionallevels of force past specific displacement depths) could be used. Withkey displacement information, a simple dedicated solenoid, motor,pneumatic, fluid, or other means can be made to have its restoring forcevary over the key travel in a highly flexible manner. Since key travelcan be fast, the transient response of the tactile feedback system musttypically have a fast rise time and be free of overshoot. Ifelectromagnetic or electric field means are used to provide keydisplacement resistance, care must be made to shield these elements toas to not create electromagnetic transients that could leak into nearbyelectronics or music instrument pickups.

[0190] Finally, it is pointed out that as an additional bonus, the abovearrangement is also capable of synthesizing different types ofmechanical so-called keyboard “actions”, for example the “feel” ofvarious types of piano manufacture keys versus harpsichord keys, etc.Thus the development of a keyboard with per-key continuous ornear-continuous displacement measurements and programmablekey-displacement resistance can provide an extraordinary level ofenhancements to conventional keyboards. This can be enhancedsignificantly with the addition of pressure sensing arrays on each keyas will be described later.

[0191] 2.1.1.4 Shared scanning electronics

[0192] In arrangements with multiple keyboards, superimposed keyboards,or related input devices (such as the strum-pads discussed below) thekeyboard-scanning electronic hardware can be in many cases largelyshared across pluralities of these keyboard contacts and/or relatedinput devices. For example, a common microprocessor could be used togenerate common multiplexing address for a group of contacts or sensorsacross several keyboards and the status of individual contacts wouldthen be serially polled or transferred in parallel.

[0193]FIG. 8 illustrates a shared scanning arrangement supporting aplurality of any of keyboards, strum-pad, buttons, switches, etc.

[0194] 2.1.2 Strum-pads

[0195] A few early music synthesizers replaced a conventional keyboardwith a low-activation pressure membrane switch array laid out toresemble a keyboard. One could freely tap or easily drag fingers overthe membrane switch array without the overhead and potential injuryinvolved in more deeply operative conventional keyboards. Because of thelack of conventional keyboard action and technique, such keyboardsrapidly lost their appeal. More recently, the Suzuki “Omnichord”product, designed to mimic an autoharp, provided a low-activationpressure membrane switch array, called a “strum-pad,” laid out to mimicthe strummed-string array of an autoharp; as a selected chord button isactivated various notes associated with the chord are assigned to thevarious membrane switches so that a finger sweeping over the strum-padproduces an arpeggiated chord in a way suggestive of strumming atraditional autoharp. The Omnichord strum-pads are hard-wired to repeatnotes multiple times and the note assignment software permits only fixedchord selections with preassigned arpeggio note sequences.

[0196] The invention includes an important element to create or expandinstruments through a generalized adaptation of these ideas:

[0197] a more generalized strum-pad element with the followingattributes:

[0198] low activation-pressure proximate switches

[0199] linear arrangement (although others are useful)

[0200] no hard-wired note repeats

[0201] visual and/or small tactile markings to the player

[0202] compact physical size

[0203] simultaneous multiple switch activation without perceivableinteraction

[0204] generalized note event information that can be assignedinterpretation under program control

[0205] more generalized strum-pad interpretation software and hardwarewith the following stored program attributes and assignments which canbe rapidly altered during playing:

[0206] assignment to selected melodic notes, percussive events, lightingor special effect events, etc.

[0207] arpeggio pattern select

[0208] note-repeats added as desired and in the manner desired

[0209] issuance of note, outgoing program change, and/or other controlsignals at the initial activation of each stored program (to sound abackground chord, activate lights, etc.) with or without activity on thestrum-pad

[0210] selection and rapid change of specific programmable attributesand assignments via button or foot-switch control

[0211] The resulting element can, for example, be attached to a guitarpick-guard and used in conjunction with foot-switches and/orfinger-activated buttons to select stored program interpretations. Freefingers can then, while freely playing the guitar as normal, “strum” ortap arpeggios, trigger percussion devices, trigger lighting or specialeffect events, etc.

[0212]FIG. 9 illustrates an example method for realizing a flexiblegeneralized strum-pad element and associated stored program control. Inthis example implementation, the strum-pad switches can be electricallywired to a simple conventional MIDI keyboard interface so that eachconsecutive switch triggers a consecutive MIDI note event. The noteevent stream is then directed to a MIDI message processor which can,under program control, reassign each incoming note event a potentiallynew MIDI note number and MIDI channel, or perhaps a null operation tocreate “safety” or “dead” zones. From here individual MIDI channels canbe directed to a variety of destinations: various synthesizer voicechannels, lighting systems, special effect systems, etc. Additionalcontrol possibilities can be further realized by translating note eventsinto other types of MIDI events, as described later, or into non-MIDIcontrol signals.

[0213] It is also possible to add note-velocity and/or“key-pressure”/“after-touch”/“channel-pressure” control to the strum-padby placing a velocity sensor (such as a piezo element) and/orpressure-sensor under it and feeding the resulting signal(s) to the MIDIkeyboard interface as would be done in a conventional MIDI keyboardrealizing these features with such sensors. It is also possible tosupplement, or replace altogether, each membrane switch with apressure-sensor, thus creating a pressure-sensor array. Such an arraycan be used to implement note-velocity and/or“key-pressure”/“after-touch”/“channel-pressure” control, but can also beused for a great many other purposes, particularly when implemented in atwo-dimensional array, as described later.

[0214] 2.1.3 Panel Controls, Actuators, Sensors

[0215] Expressive control can be enhanced considerably by attaching oneor more of any of various additional panel controls, actuators, andsensors to any electronic instrument.

[0216] Applicable types of panel controls include potentiometers (knob,slider, etc.), joysticks, panel switches, panel buttons, etc. Panelcontrols may be distributed in isolated spots, in small groups, or inarrays.

[0217] Applicable actuators can include limit switches, magneticswitches, mercury switches, optical detectors, piezo or other impactdetectors, etc. Actuators may be attached or associated with moveableparts of instruments (such as guitar vibrato “whammy” bars, harp tuninglevers, autoharp string-damper bars, etc.). Additionally, actuators maybe affiliated with the instrument as a whole, detecting rapid jarring ofthe instrument etc. Further, actuators may also be provided in isolatedspots of the instrument, such as velocity-sensitive tap-actuators forpercussion event-triggers and “body blows” to the instrument, asabstracted from for examples: ancient Chinese Pipa, centuries oldFlamenco guitar, and recent Jimi Hendrix/Adrian Belue (borderline toactual guitar abuse) techniques.

[0218] Applicable sensors can include pressure, motion (velocity,acceleration, etc.), position (optical, magnetic or electric field,electromagnetic standing wave, acoustic standing wave, etc.), impact(such as piezo sensors used with electronic drum pads), tension, strain,torsion, light, temperature, etc. Position sensors may be used tomeasure the position of a physical element of an instrument (such as adamper bar or pitch-modulating lever) or the absolute position of theinstrument itself. Tension sensors may be used, for example, to measuremodulated string tension as on a Koto or electric guitar; such stringtension controllers need not even involve sounding strings - for examplea small Koto string and bridge arrangement may be used strictly as anelectronic control provided to the player in the form of a familiar Kotostring format.

[0219] In general these panel controls, actuators, and sensors can beconfigured to provide a range of either continuous or discrete-stepcontrol voltages. In some cases additional electronics or subsequentsoftware transformations may be necessary to re-contour/redistribute thecontrol voltage over the full range of the controls, actuators, and/orsensors. In some cases, multiple transformations may be made availableunder selectable or stored program control. In any case, the resultingcontrol voltages may be then treated as generalized control signalswhich are presented to the generalized interface 110. Alternatively,some of the control voltages may be used for specialized controlsignals, such as setting values for note-velocity, after-touch, etc.

[0220]FIG. 10 shows an example implementation of both generalized andspecific control signals derived from panel controls 1001.1-1001.n,actuators 1002.1-1002.m, and sensors 1003.1-1003.k as provided for bythe invention. The panel controls 1001.1-1001.n, actuators1002.1-1002.m, and sensors 1003.1-1003.k may or may not be provided withappropriate interface electronics, respectively 1011.1-1011.n,1012.1-1012.m, and 1013.1-1013.k which deliver signals to a controlsignal formatter 1050 which issues control signals 1051. These controlsignals may be of various formats, for example MIDI.

[0221] 2.1.4 Null/Contact Touch-pads

[0222] Distinguished from panel controls and sensors considered aboveare what will be termed null/contact touch pads. This is a class ofcontact-position sensing devices that normally are in a null stateunless touched and produce a control signal when touched whose signalvalue corresponds to typically one unique position on the touch-pad.Internal position sensing mechanisms may be resistive, capacitive,optical, standing wave, etc. Examples of these devices includeone-dimensional-sensing ribbon controllers found on early Musicsynthesizers, two-dimensional-sensing pad such as the early Kawala padand more modem minipads found on some lap-top computers, andtwo-dimensional-sensing see-through touch-screens often employed inpublic computer kiosks. As a music controller these devices areattractive in that they can very easily capture very expressive fingernuances as does a violin fingerboard or Koto bridge/string arrangementbut not limit them to controlling only pitch. Two-dimensional versionsof these devices also permit the use of spatial metaphors and notions of“musical finger-painting.”

[0223] The null condition, when the pad is untouched, requires and/orprovides the opportunity for special handling. Some example ways tohandle the untouched condition include:

[0224] sample-hold (hold values issued last time sensor was touched, asdoes a joystick)

[0225] bias (issue maximal-range value, minimal-range value, mid-rangevalue, or other value)

[0226] touch-detect on another channel (i.e., a separate out-of-band“gate” channel)

[0227] Example uses for these devices as controller elements within thecontext of the invention include any one or more of the following:

[0228] issuance of melodic or percussion note events

[0229] pitch, amplitude, timbre, and location (i.e., panning, etc.)modulations

[0230] lighting and/or special effects control

[0231] general MIDI CC control signals

[0232] Additional enhancements can be added to the adaptation ofnull/contact touch pad controllers as instrument elements. A firstenhancement is, as discussed above for strum-pad elements, the additionof velocity and/or pressure sensing. This can be done via global impactand/or pressure-sensors in the same manner as described for thestrum-pads. An extreme of this is implementation of the null/contacttouch pad controller as a pressure-sensor array;

[0233] this special case and its many possibilities are described later.On the simpler extreme, a null/contact touch pad together with such aglobal velocity and/or pressure-sensor can act as a rich metaphor for adrum head, gong surface, cymbal surface, etc. and as such may be playedwith fingers, whole hands, cushioned beaters, or sticks.

[0234] A second enhancement is the ability to either discern eachdimensional-width of a single contact area or, alternatively,independently discern two independent contact points in certain types ofnull/contact controllers. FIG. 11 shows an example of how twoindependent contact points can be independently discerned, or thedimensional-width of a single contact point can be discerned, for aresistance null/contact controller with a single conductive contactplate (as with the Kawala pad product) or wire (as in a some types ofribbon controller products) and one or more resistive elements whoseresistance per unit length is a fixed constant through each resistiveelement. It is understood that a one-dimensional null/contact touch padtypically has one such resistive element while a two-dimensionalnull/contact touch pad typically has two such resistive elements thatoperate independently in each direction. Referring to FIG. 11, aconstant current source can be applied to the resistive element as awhole, developing a fixed voltage across the entire resistive element.When any portion of the resistive element is contacted by either anon-trivial contiguous width and/or multiple points of contact, part ofthe resistive element is shorted out, thus reducing the overallwind-to-end resistance of the resistance element. Because of theconstant current source, the voltage developed across the entireresistive element drops by an amount equal to the portion of theresistance that is shorted out. The value of the voltage drop thenequals a value in proportion to the distance separating the extremes ofthe wide and/or multiple contact points. By subtracting the actualvoltage across the entire resistive element from the value this voltageis normally, a control voltage proportional to distance separating theextremes of the wide and/or multiple contact points is generated.Simultaneously, the voltage difference between that of the contactplate/wire and that of the end of the resistive element closest to anextremal contact point is still proportional to the distance from saidend to said extremal contact point. Using at most simple op-amp summingand/or differential amplifiers, a number of potential control voltagescan be derived; for example one or more of these six continuously-valuedsignals:

[0235] value of distance difference between extremal contact points (or“width”; as described above via constant current source, nominalreference voltage, and differential amplifier)

[0236] center of a non-trivial-width region (obtained by simpleaveraging, i.e., sum with gain of ½)

[0237] value of distance difference between one end of the resistiveelement and the closest extremal contact point (simple differentialamplifier)

[0238] value of distance difference between the other end of theresistive element and the either extremal contact point (sum abovevoltage with “width” voltage with appropriate sign)

[0239] Further, through use of simple threshold comparators, specificthresholds of shorted resistive element can be deemed to be, forexample, any of a single point contact, a recognized contact regionwidth, two points of contact, etc., producing correspondingdiscrete-valued control signals. The detection of a width can be treatedas a contact event for a second parameter analogous to the singlecontact detection event described at the beginning. Some example usageof these various continuous and discrete signals are:

[0240] existence of widths or multiple contact points may be used totrigger events or timbre changes

[0241] degree of widths may be used to control degrees of modulation ortimbre changes

[0242] independent measurement of each extremal contact point from thesame end of the resistive element can be used to independently controltwo parameters. In the simplest form, one parameter is always largerthan another; in more complex implementations, the trajectories of eachcontact point can be tracked (using a differentiator and controlledparameter assignment switch); as long as they never simultaneouslytouch, either parameter can vary be larger or smaller than the other.

[0243] It is understood that analogous approaches may be applied toother null/contact touch pad technologies such as capacitive or optical.

[0244] A third possible enhancement is that of employing a touch-screeninstance of null/contact touch pad and position it over a video display.In this case the video display signal may be created either within aninstrument entity 100, within the signal routing, processing, andsynthesis entity 120, or from external sources such as stage cameras,attached computers, etc. The video display could for example providedynamically assigned labels, abstract spatial cues, spatial gradients,line-of-site cues for fixed or motor-controlled lighting, etc. whichwould be valuable for use in conjunction with the adapted null/contacttouch pad controller.

[0245] These various methods of adapted null/contact touch pad elementscan be used stand-alone or arranged in arrays (as in a percussioncontroller). In addition, they can be used as a component or addendum toinstruments featuring other types of instrument elements. FIG. 12 showsan example implementation of both generalized and specific controlsignals derived from electrical contact touch-pads employing MIDImessages as the output control signal format.

[0246] 2.1.5 Pressure-Sensor Array Touch-pads

[0247] The invention provides for the selective inclusion ofconsiderably advanced expressive control of electronic musical processesthrough use of a pressure-sensor array arranged as a touch-pad togetherwith associated image processing. As with the null/contact controller,these pressure-sensor array touch-pads may be used stand-alone,organized into an array of such pads, and/or used as a component and/oraddendum to instruments employing other types of instrument elements.

[0248] It is noted that the inventor's original vision of the belowdescribed pressure-sensor array touch-pad was for applications not onlyin music but also for computer data entry, computer simulationenvironments, and real-time machine control, applications to which thebelow described pressure-sensor array touch-pad clearly can also apply.

[0249] A pressure-sensor array touch-pad of appropriate sensitivityrange, appropriate “pixel” resolution, and appropriate physical size iscapable of measuring pressure gradients of many parts of theflexibly-rich human hand or foot simultaneously. FIG. 13 shows how apressure-sensor array touch-pad can be combined with image processing toassign parameterized interpretations to measured pressure gradients andoutput those parameters as control signals.

[0250] The pressure-sensor “pixels” 1300 of a pressure-sensor arraytouch-pad 1301 are interfaced to a data acquisition stage 1302. Theinterfacing method may be fully parallel but in practice may beadvantageously scanned at a sufficiently high rate to give good dynamicresponse to rapidly changing human touch gestures. To avoid the need fora buffer amplifier for each pressure-sensor pixel 1300, electricaldesign may carefully balance parasitic capacitance of the scanned arraywith the electrical characteristics of the sensors and the scan rates;electrical scanning frequencies can be reduced by partitioning theentire array into distinct parts that are scanned in parallel so as toincrease the tolerance for address settling times and other limitingprocesses. Alternatively, the pressure-sensor array 1301 may befabricated in such a way that buffer amplifier arrays can beinexpensively attached to the sensor array 1301, or the sensors 1300 maybe such that each contains its own buffer amplifier; under theseconditions, design restrictions on scanning can be relaxed and operateat higher speeds. Although the pressure-sensors may be likely analog innature, a further enhancement would be to use digital-outputpressure-sensor elements or sub-arrays. A particularly useful example ofsensor sub-arrays is presented in a few paragraphs.

[0251] The data acquisition stage 1302 looks for sensor pixel pressuremeasurement values that exceed a low-levelnoise-rejection/deformity-rejection threshold. The sufficiently highpressure value of each such sensor pixel 1300 is noted along with therelative physical location of that pixel (known via the pixel address).This noted information may be stored “raw” for later processing and/ormay be subjected to simple boundary tests and then folded intoappropriate running calculations as will be described below. In general,the pressure values and addresses of sufficiently high pressure valuepixels are presented to a sequence of processing functions which may beperformed on the noted information:

[0252] contiguous regions of sufficiently high pressure values aredefined (a number of simple run-time adjacency tests can be used; manyare known—see for example [Ronse; Viberg; Shaperio; Hara])

[0253] the full collection of region boundaries are subjected toclassification tests; in cases a given contiguous region may be splitinto a plurality of tangent or co-bordered independently recognizedregions

[0254] various parameters are derived from each independent region, forexample geometric center, center of pressure, average pressure, totalsize, angle-of-rotation-from-reference for non-round regions,second-order and higher-order geometric moments, second-order andhigher-order pressure moments, etc.

[0255] assignment of these parameters to the role of specific controlsignals (note events, control parameters, etc.) which are then output tothe signal routing, processing, and synthesis entity 120; for example,this may be done in the form of MIDI messages.

[0256] Because of the number processes involved in such a pipeline, itis advantageous to follow a data acquisition stage 1302 with one or moreadditional processing stages 1303. Of the four example processingfunctions listed above, the first three fall in the character of imageprocessing. It is also possible to do a considerable amount of the imageprocessing steps actually within the data acquisition step, namely anyof simple adjacency tests and folding selected address and pressuremeasurement information into running sums or other runningpre-calculations later used to derive aforementioned parameters. Thelatter method can be greatly advantageous as it can significantlycollapses the amount of data to be stored.

[0257] Regardless of whether portions of the image processing are donewithin or beyond the data acquisition stage, there are various hardwareimplementations possible. One hardware approach would involve verysimple front-end scanned data acquisition hardware and a singlehigh-throughput microprocessor/signal-processor chip. Alternatively, anexpanded data acquisition stage may be implemented in high-performancededicated function hardware and this would be connected to a lowerperformance processor chip. A third, particularly advantageousimplementation would be to implement a small pressure-sensor arraytogether with data equitation and a small processor into a singlelow-profile chip package that can be laid as tiles in a nearly seamlesslarger array. Such “mini-array” chips have additional value as they canreadily be put on instrument keys (as described below), instrumentfingerboards, instrument bodies, etc. In such an implementation allimage processing could in fact be done via straightforward partitionsinto message-passing distributed algorithms. One or more individualchips could direct output parameter streams to an output processor whichwould organize and/or assign parameters to output control channels,perhaps in MIDI format, perhaps in a programmable manner underselectable stored program control. A tiled macro array of such “sensormini-array” chips could be networks by a tapped passive bus, one- ortwo-dimensional mode active bus daisy-chain, a potentially expandablestar-wired centralized message passing chip or subsystem, or othermeans. Creating a large surface from such “tile chips” will aid in theserviceability of the surface. Since these chips can be used as tiles tobuild a variety of shapes, it is therefore possible to leverage asignificant manufacturing economy-of-scale so as to minimize cost andjustify more extensive feature development. Advanced seating andconnector technologies, as used in lap-tops and other high-performanceminiature consumer electronics, can be used to minimize the separationbetween adjacent chip “tiles” and resultant irregularities in thetiled-surface smoothness. A tiled implementation may also include a thinrugged flexible protective film that separates the sensor chips from theoutside world. FIG. 14 illustrates the positioning and networking ofpressure sensing and processing “mini-array” chips in both largercontiguous structures and in isolated use on instrument keys, instrumentfingerboards, and instrument bodies.

[0258] With the perfection of a translucent pressure-sensor array, itfurther becomes possible for translucent pressure-sensor arrays to belaid atop aligned visual displays such as LCDs, florescent, plasma,CRTs, etc. as was discussed above for null/contact touch-pads. Thedisplays can be used to label areas of the sensor array, illustrategradients, etc. Note that in the “tile chip” implementation, monochromeor color display areas may indeed be built into each chip.

[0259] Returning now to the concept of a pressure-sensor array touch-padlarge enough for hand-operation: examples of hand contact that may berecognized, example methods for how these may be translated into controlparameters, and examples of how these all may be used are now described.In the below the hand is used throughout as an example, but it isunderstood that the foot or even other body regions, animal regions,objects, or physical phenomena can replace the role of the hand in theseillustrative examples.

[0260]FIG. 15 illustrates the pressure profiles for a number of examplehand contacts with a pressure-sensor array. In the case 1500 of afinger's end, pressure on the touch pad pressure-sensor array can belimited to the finger tip, resulting in a spatial pressure distributionprofile 1501; this shape does not change much as a function of pressure.Alternatively, the finger can contact the pad with its flat region,resulting in light pressure profiles 1502 which are smaller in size thanheavier pressure profiles 1503. In the case 1504 where the entire fingertouches the pad, a three-segment pattern (1504 a, 1504 b, 1504 c) willresult under many conditions; under light pressure a two segment pattern(1504 b or 1504 c missing) could result. In all but the lightestpressures the thumb makes a somewhat discernible shape 1505 as do thewrist 1506, cuff 1507, and palm 1508; at light pressures these patternsthin and can also break into disconnected regions. Whole hand patternssuch the fist 1511 and flat hand 1512 have more complex shapes. In thecase of the fist 1511, a degree of curl can be discerned from therelative geometry and separation of sub-regions (here depicted, as anexample, as 1511 a, 1511 b, and 1511 c). In the case of the whole flathand 1500, there can be two or more sub-regions which may be in factjoined (as within 1512 a) and/or disconnected (as an example, as 1512 aand 1512 b are); the whole hand also affords individual measurement ofseparation “angles” among the digits and thumb (1513 a, 1513 b, 1513 c,1513 d) which can easily be varied by the user.

[0261] Relatively simple pattern recognition software can be used todiscern these and other hand contact patterns which will be termed“postures.” The pattern recognition working together with simple imageprocessing may, further, derive a very large number of independentcontrol parameters which are easily manipulated by the operating user.In many cases it may be advantageous to train a system to theparticulars of a specific person's hand(s) and/or specific postures. Inother situations the system may be designed to be fully adaptive andadjust the a persons hand automatically. In practice, for the widestrange of control and accuracy, both training and ongoing adaptation maybe useful. Further, the recognized postures described thus far may becombined in sequence with specific dynamic variations among them (suchas a finger flick, double-tap, etc.) and as such may be also recognizedand thus treated as an additional type of recognized pattern; suchsequential dynamics 5 among postures will be termed “gestures.” Theadmission of gestures further allows for the derivation of additionalpatterns such as the degree or rate of variation within one or more ofthe gesture dynamics. Finally, the recognized existence and/or derivedparameters from postures and gestures may be assigned to specificoutgoing control signal formats and ranges. Any training informationand/or control signal assignment information may be stored and recalledfor one or more players via stored program control.

[0262] For each recognized pattern, the amount of information that canbe derived as parameters is in general very high. For the human hand orfoot, there are, typically, artifacts such shape variation due toelastic tissue deformation that permit recovery of up to all six degreesof freedom allowed in ,an object's orientation in 3-space. FIG. 16illustrates how six degrees of freedom can be recovered from the contactof a single finger. In the drawing, the finger 1600 makes contact withthe touch-pad 1601 with its end segment at a point on the touchpadsurface determined by coordinates 1611 and 1612 (these would be, forexample, left/right for 1611 and forward/backward for 1612). Fixing thispoint of contact, the finger 1600 is also capable of rotational twistingalong its length 1613 as wall as rocking back and forth 1614. The entirefinger can also be pivoted with motion 1615 about the contact pointdefined by coordinates 1611 and 1612. These are all clearlyindependently controlled actions, and yet it is still possible in anyconfiguration of these thus far five degrees of freedom, to vary theoverall pressure 1616 applied to the contact point. Simple practice, ifit is even needed, allows the latter overall pressure 1616 to beindependently fixed or varied by the human operator as other parametersare adjusted.

[0263] In general other and more complex hand contacts, such as use oftwo fingers, the whole hand, etc. forfeit some of these example degreesof freedom but often introduce others. For example, in the quiteconstrained case of a whole hand posture, the fingers and thumb canexert pressure independently (5 parameters), the finger and thumbseparation angles can be varied (4 parameters), the finger ends 1504 acan exert pressure independently from the middle 1504 b and inner 1504 csegments (4 parameters), the palm can independently vary its appliedpressure (1 parameter) while independently tilting/rocking in twodirections (3 parameters) and the thumb can curl (1 parameter), yielding17 instantaneously and simultaneously measurable parameters which areindependently adjustable per hand. Complex contact postures may also beviewed as, or decomposed into, component sub-postures (for example here,as flat-finger contact, palm contact, and thumb contact) which wouldthem derive parameters from each posture independently. For such complexcontact postures, recognition as a larger compound posture which maythen be decomposed allows for the opportunity to decouple and/orrenormalize the parameter extraction in recognition of the specialaffairs associated with and constraints imposed by specific complexcontact postures.

[0264] It is noted that the derived parameters may be pre-processed forspecific uses. One example of this would be the quantization of aparameter into two or more discrete steps; these could for example besequentially interpreted as sequential notes of a scale or melody.Another example would be that of warping a parameter range as measuredto one with a more musically expressive layout.

[0265] Next examples of the rich metaphorical aspects of interactingwith the pressure-sensor array touch-pad are illustrated. In many casesthere may be one or more natural geometric metaphor(s) applicable, suchas associating left-right position, left-right twisting, or left-rightrotation with stereo paning, or in associating overall pressure withvolume or spectral complexity. In more abstract cases, there may bepairs of parameters that go together—here, for example with a fingerend, it may be natural to associate one parameter pair with (left/rightand forward/backward) contact position and another parameter pair with(left/right and forward/backward) twisting/rocking. In this latterexample there is available potential added structure in the metaphor byviewing the twisting/rocking plane as being superimposed over theposition plane. The superposition aspect of the metaphor can be viewedas an index, or as an input-plane/output-plane distinction for atwo-input/two-output transformation, or as two separated processes whichmay be caused to converge or morph according to additional overallpressure, or in conjunction with a dihedral angle of intersectionbetween two independent processes, etc.

[0266] Next, examples of the rich syntactical aspects of interactingwith the pressure-sensor array touch-pad are illustrated. Someinstruments have particular hand postures naturally associated withtheir playing, particularly hand drums and especially Persian and Indianhand drums (such as the tabla/baya bols, dumbek, etc.). It is naturalthen to recognize these classical hand-contact postures and derivecontrol parameters that match and/or transcend how a classical playerwould use these hand positions to evoke and control sound from theinstrument. Further, some postures could be recognized either inisolation or in gestural-context as being ones associated with (orassigned to) percussion effects while remaining postures may beassociated with accompanying melodies or sound textures. As anadditional syntactic aspect, specific hand postures and/or gestures maymapped to specific selected assignments of control signals in waysaffiliated with specific purposes. For example, finger ends may be usedfor one collection of sound synthesis parameters, thumb for a secondpotentially partially overlapping collection of sound synthesisparameters, flat fingers for a third partially-overlapping collection,wrist for a fourth, and cusp for a fifth, and fist for a sixth. In thiscase it may be natural to move the hand through certain connectedsequences of motions; for example: little finger end, still in contact,dropping to flat-finger contact, then dropping to either palm directlyor first to cusp and then to palm, then moving to wrist, all neverbreaking contact with the touch-pad. Such permissible sequences ofpostures that can be executed sequentially without breaking contact withthe touch-pad will be termed “continuous grammars.” Under thesecircumstances it is useful to set up parameter assignments, andpotentially associated context-sensitive parameter renormalizations,that work in the context of selected (or all available) continuousgrammars. For example, as the hand contact evolves as being recognizedas one posture and then another, parameters may be smoothly handed-overin interpretation from one posture to another without abrupt changes,while abandoned parameters either hold their last value to return to adefault value (instantly or via a controlled envelope).

[0267] Now a number of example applications of the pressure-sensor arraytouch-pad are provided. A natural start for a first example is that ofthe Indian tabla and baya; here the traditional bols are recognized andused to control synthesized or sample-playback sound generation.

[0268] The produced sound can be authentic or transcend the classicalinstrument. Additional posture and gesture recognition can be added ineither sound generation style to expand the available sounds and/orcontrol additional signal processing such as location modulation,muffling or peaking filtering, reverb, sustain, instrument pitch, etc.Considering hand drums more generally it is noted that whole-hand slapsare commonly used in the technique but that the spread of the fingers inthe hand slap or hand after-touch of the drum head typically provide nousable control. With the system described above, details of at leastfour parameters of finger spread and even more on whole-hand posture inwhole-hand slaps and ongoing after-touch pressing may be used forextensive timbre variation.

[0269] Next, examples are given as to how derived parameters may be usedto control musical processes and lighting control, effectively allowingone to “fingerpaint” with sound and/or light. There are a large numberof ways in which six parameters of synthesizer “voices” may becontrolled with one finger. One possible example of a mapping is to useall six parameters to control prominent features of a single synthesizervoice:

[0270] left/right position: pitch

[0271] in/out position: volume

[0272] left/right twist: waveform morphing dimension 1 (“duty cycle,”even-harmonic content, etc.)

[0273] in/out rock: waveform morphing dimension 2 (“waveform curvature,”odd-harmonic content, etc.)

[0274] rotation: stereo pan

[0275] overall pressure: filter opening

[0276] Another example is that of controlling two voices with onefinger:

[0277] left/right position: pitch of voice 1

[0278] in/out position: pitch of voice 2

[0279] left/right twist: pan or filter opening of voice 1

[0280] in/out rock: pan or filter opening of voice 2

[0281] rotation: relative volume balance of voice pair

[0282] overall pressure: total volume of voice pair

[0283] By assigning pitch to an aspect of physical contact that isgeometrically large (i.e., position on the pad), it is possible to get agreat deal of accuracy in pitch control. In potentially typically caseswhere pitch choices are to be associated with traditional scales, thepitch control parameter may be quantized into discrete steps and eachstep assigned to a note in a scale or melody. At the point of contactwithin a selected quantization interval, a small “vibrato” neighborhoodmay then be defined so that wiggling the finger position is mapped to avibrato-range pitch variation (as on a violin string). If thespatially-quantized positions are mapped to notes in a melody, it ispossible to set up mappings for several musical phrases or in fact anentire melodic line start-to-finish. In the latter circumstance, it maybe desirable to either “page” the pitch assignments to give up one ofthe position parameters for sound control or instead use it for layingout the melody geometrically as per a sheet of music; here the spatialquantization may be uniformly spaced or under limited degrees beproportional to the pitch duration of the associated note. Thesheet-music layout is particularly interesting because it allows theperformer to concentrate extreme dexterity in the timbre and timingexpression of a melody without having to devote very much effort orattention to the selection of pitch value. The resulting allocationshift of performer attention is very valuable as the amount ofexpression and variations in timbre are often what distinguish aspellbinding performance from a run-of-the-mill performance. Althoughpurist musicians may scoff at the release from pitch selection strugglesendemic in musical instruments over the centuries, they are also knownto spend thousand of dollars on finest-instruments that allow additionalnuances of expression and spend many, many years of their lives makingpitch selection efforts nearly as subconscious as this instrumentapproach does. This class of instrument controller, then, allows thoseyears of skill development to be devoted directly to perfecting advanceddegrees of musical expression, potentially higher than may be achievedwith conventional human life spans, traditional real-time instruments,and orchestra-conductor protocol.

[0284] Leaving higher callings in music for the moment, it is alsopossible to use the pressure-sensor array touch-pad for lightingcontrol, particularly multi-channel lighting and/or motor-controlled(any one or more of pan, tilt, zoom, gel, pattern-gel orientation, etc.)lighting. In multiple-light control situations, regions of the pad maybe quantized into cells, each associated with a particular light andparameters within the region, controlling any of: light, brightness,position, zoom, gel, gel-pattern-orientation, etc. What can beespecially interesting in performance is to combine music processcontrol with lighting control. Some postures, gestures, or pad-regionsmay be exclusively devoted to only music control or only lightingcontrol parameters, but other postures, gestures, or pad-regions may beset up to intermingle and/share parameter assignments between music andlights.

[0285] It is also known to be possible and valuable to use theaforementioned pressure-sensor array touch-pad, implicitly containingits associated data acquisition, processing, and assignment elements,for many, many non-musical applications such as general machine controland computer workstation control. One example of machine control is inrobotics: here a finger might be used to control a hazardous materialrobot hand as follows:

[0286] left/right position: left/right hand position

[0287] in/out position: in/out hand position

[0288] in/out rock: up/down hand position

[0289] rotation: hand grip approach angle

[0290] overall pressure: grip strength

[0291] left/right twist: gesture to lock or release current grip frompressure control

[0292] A computer workstation example may involve a graphicalComputer-Aided Design application currently requiring intensive mousemanipulation of parameters one or two at a time:

[0293] left/right position: left/right position of a selected symbol ina 2-D CAD drawing

[0294] in/out position: up/down position of a selected symbol in 2-D CADdrawing

[0295] left/right twist: symbol selection-left/right motion through 2-Dpallet

[0296] in/out rock: symbol selection -up/down motion through 2-D pallet

[0297] rotation: rotation of selected symbol in the drawing

[0298] overall pressure: sizing by steps

[0299] tap of additional finger: lock selection into drawing or unlockfor changes

[0300] tap of thumb: undo

[0301] palm: toggle between add new object and select existing object

[0302] Clearly a symbol can be richly interactively selected andinstalled or edited in moments as opposed to tens to hundreds of secondsas is required by mouse manipulation of parameters one or two at a timeand the necessary mode-changes needed to change the mouse actioninterpretation.

[0303] 2.1.6 Multi-Parameter Instrument Keys

[0304] The famous multiple tape-loop Melletron product had keys whichserved to a rough extent as per-note volume controls, allowing valuablerelative voice level variations. Robert Moog patented a key with atwo-dimensional touch sensor on a keyboard key surface. The presentinvention allows for the synergistic combination of these technologiesso as to create a three-parameter controlling key particularly suited tovowel-choir synthesis and other applications, next extends this toinclude more arbitrary instrument keys (such as those on a woodwind),and finally develops multi-parameter sensing keys further byincorporation of the aforementioned pressure-sensor array touch-pad oneach key.

[0305] When voice choirs are used as instrumentation rather than thedeliverer of libretto, the principal parameters are typically the vowelsound used and the relative amplitude of each vocal line. If theseparameters were to be controlled by a keyboard, and for the moment ifunisons of two or more vocal lines were excluded (unisons will in factbe handled later), each vocal line would be at a different pitch fromthe others. This allows at any particular instant specific keys on akeyboard to be uniquely associated with one vocal line apiece. As withthe now somewhat traditional Melletron, the displacement of key soundingthe note of a particular vocal line then may be used to control thevolume of that vocal line. By incorporating a two-dimensional touch-padcontroller to each key, it is also possible to select and in fact varythe vowel sound. In phonetics and vocal pedagogy it is well known[Appelman, Winckel] that the quality of the vowel is largely determinedby the frequencies of resonances produce by the vocal cavity. In fact,the full range of realistic vowel sounds may be created by passingsimple sawtooth or narrow-wide pulse oscillator waveforms into a pair ofband emphasis filters, the vowel sounds varying as the filter emphasisfrequencies are varied. FIG. 17, adapted from Winckel, illustrates theregions of vowel sounds associated with particular resonant frequencycombinations in vowel sound production. Clearly there are twodimensions, then, which control vowel quality at this level ofapproximation, and further the surface of the key may be viewed as ametaphor for the plot of FIG. 17. Further details of effective choirsynthesis and variations upon it are discussed later, but thesynergistic value of the two-dimensional touch-pad key surface and keydisplacement as sources of control signals for choir synthesis isclearly established. In fact, this three-parameter per individualsynthesizer “voice” may be very valuable in at least two additionalsituations.

[0306] In a first of these additional situations, it is first noted thatin traditional multiple-instrument orchestration, the principalparameters are volume and timbre. Using the aforementionedthree-parameter key arrangement, key displacement may again be used forper-note volume control, leaving the remaining two dimensions for timbrecontrol. Wessel and others have shown empirically that continuousmultidimensional “timbre spaces” are useful organizations for analyzingand executing orchestration aspects of timbre assignment. Oftentwo-dimensional timbre spaces offer a more than rich enough environmentto be very useful. FIG. 18, adapted from Wessel, illustrates someexample two-dimensional timbre spaces from traditional instrumentorchestration. Again, a metaphor may be made between thesetwo-dimensional graphs and the two-dimensional touch-pad key surface.The implementation on the synthesis side may be implemented by methodsas simple as volume cross-fading of sampled traditional instruments(and/or synthesized sounds) to methods as sophisticated as morphablenumeric instrument models.

[0307] The second additional situation also pertains to so-calledmodel-based synthesis (as employed in the Yahama VL1) but over a lesserrange of timbre variation, in fact a range typically within the scopeassociated with a single instrument rather than a multi-instrumentorchestration environment. Model-based synthesis typically has anabundance of parameters and a dearth of effective methods forcontrolling them. Selected parameters, in fact, are controlled withglobal control interfaces such as a wheel, joystick, or breathcontroller. Because of the need for associating parameter control witheach note, rather than a group of notes, the best model-based synthesisengines then have been monophonic (i.e., only producing one note at atime). The invention's provision of a keyboard with the availability ofthree parameters tied specifically and independently to each key is anideal solution to a polyphonic model-based synthesis instrument.

[0308] It is noted that choir synthesis, dynamic timbre-space basedorchestration, and polyphonic model-based synthesis instruments requirethe synergistic combination of key displacement and key surfacetouch-pad, while the said combination also is fully capable ofimplementing Moog's original vision for two-dimensional synthesiscontrol (filter parameters, oscillator waveforms, etc.) and as analternative implementation to MIDI keyboard channel pressure whichtypically requires each active key to be fully displaced.

[0309] The invention also provides for the application control discussedabove to be enhanced yet further by placing a pressure-sensor arraytouch-pad on each key. In the limit, this would allow each key to deriveup to six parameters for each point of contact on a key and evenmultiple points of contact (i.e., more than one finger) per key.Although custom pressure-sensor array touch-pads could be crafted forthe keys, it is advantageous to employ the aforementioned pressuresensing and processing “mini-array” chips. In fact, applications to keysurfaces could be used to dictate the canonical dimensions of the chips,for example the width of the top surface of a black key and a lengththat is a least common multiple of a black key surface length and awhite key surface length.

[0310]FIG. 19 shows an example of keys from a traditional Westernkeyboard fitted with multiple uniformly-sized pressure-sensing andprocessing “mini-array” chips. These chips may be interconnected usingthe networking features described earlier. Alternatively, specialpressure-sensing and processing “mini-array” keys may be made withoutthe chip as a tiling sub-component; these would be networked in the samefashion. As to how the uniformly-sized pressure-sensing and processing“mini-array” chips could be applied to the keys, FIG. 19 first shows thecollection of key shapes 1900 used to make a conventional Westernkeyboard. There are five types of white key shapes (1901-1905) and oneblack key shape 1900 used, although the black key 1900 may have a taperleading from its widest base area 1906 a to a narrower top area 1906 b.Each of these white keys may be viewed in general terms as thecombination of two adjoined areas: one forward area rectangle ofdimensions 0.75″ by 1.75″ and a rear area bar typically at least 0.375″wide (varying with different styles of keys). One example black key hasa taper leading to a top surface 1906 b of 0.25″ wide by 2″ deep, andanother example black key has a lesser taper with a top surface 1906 bhat is 0.3125″ wide by 2″ deep. As one example, a sensor size of 0.25″wide by 0.58″ deep could be tiled on the six keys 1901-1906 according tothe arrangement of 1910. As another example, a sensor size of 0.25″ wideby 0.75″ deep could be tiled on the six keys 1901-1906 according to thearrangement of 1920. As a third example, a sensor size of 0.75″ deepcould be tiled on the six keys 1901-1906 according to the arrangement of1930.

[0311] A point not discussed yet-though relevant to all thecontrollers-is one that is especially relevant to all forms ofmulti-parameter touch-sensing keys: that is the perceptual trade-offbetween note duration and the perception of timbre detail. In shortduration notes the ear is not able to gather much information about thetimbre of the note, while in long notes the ear typically examines thetimbre, as well as any inherent harmonic animation therein, inconsiderable detail and becomes easily turned away when there is novariation, or easily learned predictable variation, in harmonic contentover time. The multi-parameter touch-sensing keyboard is thuswell-targeted for this phenomenon in hearing. On rapid notesmulti-parameter touch-sensing keys may actually be played withincreasing degrees of timbre-control arbitrariness, while longer notesmay be played with a great deal of timbre and amplitude variation.Although two degrees of freedom afford by the Moog key is helpful inadding per-voice expression for long-duration notes, the three degreesof freedom provided by the aforementioned techniques in practice seemsto be a minimal control-dimensionality threshold for useful musicalexpression. A venture as to why two parameters are not enough couldstart with the fact that there is great importance in relative volumevariation between voices—this leaves only one parameter then for timbervariation which quickly bores the ear; adding another dimension allowsfor more sophisticated temporal interplays and variations over time intimbre qualities. Empirical support for this is seen in the fact thatdiscussions of “timbre space” and “sound color” in the literature devotea minimum of two-dimensions to timbre. A venture as to why the interplayof two timbre dimensions itself is a minimal control-dimensionalitythreshold for timbre could resort to an abstraction of FIG. 17: humanhearing is attuned to speech which is in turn a sequence ofphonemes—each phoneme, roughly, is a vowel sound modulated in timeaccording to some consonant aspects and supplemented by, loosely,“percussive” effects in other consonant aspects. With respect to timbrein phonemes, the ability of human hearing to follow and distinguishvowels and their modulation (including diphthongs) is largely centeredon essentially independent variation in the two formats. Thus, it couldbe postulated that amplitude variation together with two-dimensionaltimbre variation engages the speech center of the brain in a full andnatural way. As with speech, words and phonemes spoken quickly cannot bediscerned with nearly as much expression as words and phonemes spoken inlong duration can, and in long duration phonemes the ear is pleased withexpressive fluctuations in timbre and amplitude.

[0312] Finally, as to the handling of unisons (and the related problemof melodic line pitch crossings of uncommon timbre), in usual practice(and prior to the invention) these are typically addressed by use ofmultiple keyboards or by a split of keyboard ranges into independentlyinterpreted zones. The addition of proximate keyboards and superimposedkeyboards as afforded by the invention significantly enhances thepractical extent to which and ease by which unisons and melodic linepitch crossings may be handled. As a simple example, if all melodiclines have timbre ranges that lie in a common range, and unisons sharingthe same timbre unisons may be naturally handled by superimposedkeyboard aspect of the invention—push the key deeper, or harder, for two(or sequentially, three, four, etc.) notes in unison all following thesame timbre control. More generally, proximate keyboards may be used topartition the notes that may be played with one hand between two, and insome cases three, distinct keyboards; this freely allows the player, inall but some pathological cases, to independently control unisons andmelodic line pitch crossings without constraint as to relative timbredifferences.

[0313] 2.1.7 Video Cameras and other optically-controlled sensors

[0314] Video cameras and other optically-controlled sensors may also beused as control elements within an instrument 100. As with otherinstrument elements, video cameras and other optically-controlledsensors may be used stand-alone, in arrays, or as component/addendum toother instruments. Video cameras are especially interesting ascontrollers because of available image processing, image recognition,and image motion tracking utilities which have been developed formanufacturing inspection, medicine, and motion-video compressiontogether with the ability to actually display a real-time image inrecording or performance.

[0315] 2.1.7.1 Non-video optically-controlled sensors

[0316] So as to devote most of the discussion to video, the case ofsimple non-video optically-controlled sensors is first considered. Asimple example is a set of photo-detectors which are used to discretelytrigger one or more note, lighting, or special effect events. Forexample, a light-harp without strings may trigger notes, potentiallytogether with selected stage lights and artificial fog blasts, as thefingers interrupt light beams directed towards the photo-detectors.Another example is that of a stage area with an array of light beamsdirected towards an associated first group of photo-detectors: the beamsto individual photo-detectors of this first group may be interrupted, orredirected by means of reflective surfaces to a second group ofphoto-detectors, by dancers, actors, or musicians in choreographedmovement; the various deactivations and activations of photo-detectors,respectively, may trigger one or more of: note, lighting events, orspecial effects. It is noted that a later described aspect of theinvention provides for the generation of an event base on the detectionof predefined sequences of events; here then certain note phrases orpaths through the stage installation would trigger additional eventssuch as fog blasts illuminated by selected colors of light which aredistinguished by the pattern detected.

[0317] A more sophisticated use of simple non-video optically-controlledsensors is to continuously control one or more of sound, lighting, orspecial effect parameters; here the photo-detection is not one of on/offon a relatively narrow beam but rather continuous intensity variation ofa relatively wider light beam. The light intensity directed at aphoto-detector may be varied by means of varying the percentage of lightinterruption by the parts of the human body, clothing, artificial fogclouds affected by a performer, or other translucent, light-reflectiveor light-refractive objects manipulated by a performer.

[0318] In the above, the source light may exist in an environment ofperformance stage lighting or other illumination. To limit interferenceon the instrument, light sources may be any one or more offrequency-modulated, selected-wavelength operation, or minimum-intensityoperation (via inexpensive low-power lasers) methods. Alternatively, orin addition, a photodetector may be provided with anoptically-directional shroud to limit interfering ambient light.

[0319] It is also possible to actually use stage lights as light sourcesfor photo-detection as an aspect of the invention. For example, aspotlight beam may be directed, via light-reflective or light-refractiveelements operated by performers, on to one or more photo-detectorsoperating in either discrete-trigger or continuous-variation modes.

[0320] Finally, it is possible for the photo detectors to be colorsensitive. This may be done any number of ways, ranging from puttingcolor filters over photo-detectors to using color electronic cameras andsimple image processing to derive average measured color. Should acamera be used for color or other photo-detection roles, photo-detectorsites may actually be fiber optic paths that lead to a centralizedcamera element. Light color directed to the photo-detectors may bevaried by performers by means of filters, prisms, or other manipulabletranslucent, reflective, of refractive objects.

[0321] 2.1.7.2 Video cameras

[0322] Video cameras may be attached to an instrument for showingclose-up of the performer's playing. The video close-up feed may bedisplayed on monitors during a performance or recorded, and as discussedlater, potentially involving other video sources and potentially with orwithout special effects. For movable instruments, such as guitars,woodwinds, etc. this can create an interesting visual effect as theinstrument profile will be firmly fixed in the video image while theambient visual background will move as the performer moves theinstrument. These visual effects seem to work best with instruments thathave sufficient physical inertia and/or which are supported by straps;instruments subject to significant undamped motion, such as flutes, mayactually have so much background motion that the image is uncomfortableto watch.

[0323] Video cameras, be they attached to an instrument or not, may alsobe used as instrument elements by processing the video image signals todegrees that range from simple average image brightness calculationthrough pattern recognition to image interpretation. In a simpleexample, the luminance signal for each video frame or interlace-field(i.e., only the odd or only even lines) may be sent to an integraterelement followed by a sample-hold element;

[0324] the integrator may be further enhanced to not integrate duringretrace intervals. The result gives the average brightness of theprocessed image. Adding two such additional integrate/hold elements andfeeding the three the red/green/blue decomposition of a color videosignal makes an image-averaged color detector. In these ways the samecamera that produces performance and/or recording video images may beused as a non-video optical sensor in the manners described earlier.This primitive capability, then, may allow a performer to tilt or rotatethe instrument 100 position so as to include stage lights or backgroundimages of particular brightness and/or colors, direct or impede incominglight with the hand or objects, cover the lens, etc., and in so doingtrigger and/or continuously control sound, lighting, or special effectevents. The latter may occur when the video image is being displayedand/or recorded or with the video signal used solely in an instrumentmode.

[0325] Far more valuable is the use of the spatial capture aspects of avideo camera. A simple example of this would be to split the image into“sub-image cells” (i.e., half, quarters, etc. of the entire video image)using various means and again deriving average luminance and/or colorinformation from each of the cells. For small numbers of cells this maybe done with analog electronics: sync detectors trigger one-shots thatgate specific integrate/hold circuits for specific intervals ofhorizontal scan lines in specific vertical regions of the image. Digitalmethods may also be used, for example: reading the image into a framebuffer which is then analyzed in the retrace interval for the nextframe, doing running calculations on the video signal as the fields arescanned, etc. Digital methods will typically scale to higher resolutionsand more complex functionalities and thus in many cases may bepreferred. Digital methods may be implemented with special dedicatedhardware or standard personal computers fitted with standard videocapture and MIDI interface cards, etc. Such personal computerimplementations may implement a number of image processing, parameterderivation, and control signal assignments in a flow virtually identicalto that of FIG. 13. These functions may be done in software run on thepersonal computer or in part or in whole by dedicated hardware boards orperipherals (for functions such as video acquisition, patternrecognition, etc.)

[0326] With the ability to process images at higher resolutions and inmore complex ways, it becomes possible to use video in increasinglyvaluable ways as an instrument element. By correlating higher resolutionimage area measurements, it becomes possible to recognize patterns andshapes and derive parameters from them in real-time. In fact, the sameimage processing software structures used in pressure-sensor arraytouch-pads, or even exact portions of software itself, may also be usedto process video images in real-time, replacing pressure pixelinformation with, for example, luminance pixel information. Thesealgorithms may be enhanced further by exploiting available colorinformation as well. The shapes recognized and some of the parametersderived from them are likely to have a somewhat different quality: the3D-projected-to-2D nature of camera images, gradients of luminancecreated by shadows and reflections, as well as the types and(potentially) ranges of shapes to be recognized typically differsignificantly from those discussed in the pressure-sensor arraytouch-pad context. Nevertheless, similar software structures may be usedto great value. Specific types of shapes and patterns-such as writtencharacters, particular gradients in brightness or color, separationdistances between bars and/or bar widths-may be particularly usefulvariations from those shapes and patterns discussed in the context ofpressure-sensor array touch-pads.

[0327] Next: examples of how video cameras supplemented with thesecapabilities may be used to trigger events and/or continuously controlsound, light, and special effects.

[0328] A first example is that of recognizing the human hand posture,position, and proximity to the camera in 3-space. Simple handorientation and posture geometry may be used to create specific controlsignals. In a more advanced implementation, dynamic gestures may berecognized. These two capabilities give the system, with sufficientsoftware, the ability to recognize a few if not many verbal handsignals; with yet more enhancements, potentially including the abilityto recognize the roles of two hands with respect to the human body, therecognition capabilities could include, for example, formal ASL as wellas particular dance postures. The ability to recognize postures of hand,hand/arm, hand/arm/body, etc. allows hands, dance, “conducting” (notnecessarily restricted to formal conducting gestures), etc. to be useddirectly for the control of sound, lighting, and special effects.

[0329] In another class of examples, video cameras may recognize, andderive parameters from, characters and/or patterns available on a stage.Such characters and/or patterns may be brought before the camera,exposed and obfuscated from the camera; the camera may be turned towardsthe characters and/or patterns, etc., resulting in derived parametersand issued control signals. Stage cameras may also be used to recognizeand track the location and some aspects of body orientation and postureof performers, deriving parameters and issuing control signals fromthese as well.

[0330] In each of the above examples, it is noted that the use of two ormore cameras, either in stereoscopic layout similar to those of humaneyes or in an orthogonal layout (i.e., forward facing camera andoverhead camera covering the same 3-space region), may be used toresolve 3D-to-2D projection singularities in the pattern and shaperecognition and processing.

[0331] As a third class of example, recent developments have allowed forthe recognition of human facial expressions from video images and evendegrees of lip reading. These recognition and parameter derivationmethods may also be adapted in the invention to provide the ability forthe human face to be used as a controller for sound, lighting, andspecial effects. Simplified systems can be created to recognized andparameterize a few selected expressions or to recognize and measuregeometric variations in specific areas of the face.

[0332] From a formal, traditional music perspective, much of the abovemay appear to be gimmickry with meaningful application at best in avantgarde installations or modern play products. In one response to this,directed on hand posture capture, it is noted that the hand in 3-spaceis clearly the most physically expressive aspect of the human body andis used to control almost all musical instruments but by very restrictedgeometric means. Freeing the hand to move unrestricted allowsconsiderably more expression to be captured. Further then, as a fourthexample, advances in cost reductions for video cameras and signalprocessing can make it possible for an array of cameras to be devoted toa traditional instrument controller, such as a keyboard, drum head, orflute key array (as well as, for example, a pressure-sensor arraytouch-pad) so as to capture hand expressions that cannot otherwise becost-effectively captured from the instrument controller.

[0333] Final, a brief preliminary discussion is provided here on thesignificant role of video in compositional and performance semiotics.For many years music, dance, art, film, plays, literature, poetry,linguistics, and other fields have come under study and compositionalmethods involving common abstractions or “signs” that live within andamong their works and idioms. More will be said later about theinvention as a whole as an environment for more significantly exploitingsemiotics as a compositional and performance tool. However, videocameras used as an instrument element, either with or without the videostream being displayed or recorded, offer a special role in the creationof semiotic elements because they may be used to link visual symbols ofobject and body to sound, lighting, and special effects which in turnmay have assigned and/or intrinsic semiotic content.

[0334] 2.1.8 Singing and Speech Detection, Recognition, andParameterization

[0335] Speech recognition systems have become increasing accurate andinexpensive. These technologies can, in many valuable ways, be adaptedto also recognize sung words and/or phonemes. Recognized words orphonemes may be used to trigger any of sound, lighting, or specialeffect events, while existing pitch detection and amplitude followingtechnologies (as found, for example, in the early Roland CP-40 productor in the more modem MidiVox SynchroVoice product) may be used to derivecontinuous control signals. In addition, interevent timers may be usedto measure individual word and/or phoneme duration.

[0336] These singing and speech recognition capabilities together withtheir parameterization also have significant potential value in theaforementioned creation of semiotic elements because they can be used tolink verbal linguistic events and expression to sound, lighting, andspecial effects which in turn may have assigned and/or intrinsicsemiotic content.

[0337] 2.1.9 Air Pressure, Air flow, and Air Turbulence Sensors andTransducers

[0338] Air flow, or “breath,” controllers for musical instruments areknown and have been employed in electronic woodwind-like controllers. Itis a provision of the invention to include these along with air pressureand air turbulence sensors and transducers as elements of an instrumententity 100. In particular, air pressure-sensors can be attached to airbladders to form a particular kind of pressure or squeezing controller.Air pressure-sensors can also be introduced into a wind instrumentinterior in an instrument where subsonic variations in ambient pressureoccur as the instrument is played.

[0339] Traditional wind instrument players often invoke air turbulenceeffects, such as transient “chiffs”, tongue trills, overblowing, etc.Air turbulence is then also a candidate control interface for use in anelectronic instrument entity 100. Air turbulence sensors may be craftedin various ways, including by means of signal processing the output ofany one or more of air flow and/or air pressure-sensors. A simpleexample would be to define a high-pass cutoff frequency for air flowand/or air pressure variations and another (higher) low-pass cutofffrequency for the lowest musical “pitched” frequencies; the energy inthe remaining band of frequencies would be a crude measure or airturbulence. In a more sophisticated implementation, an array of airpressure-sensors can be distributed throughout a wind tube andsensor-array signal processing techniques can be used to separateturbulence signals from environmental acoustic noise, standing waves inthe tube, etc.

[0340] 2.1.10 Clothing, Jewelry, Skin, and Muscle Sensors

[0341] Sensors on the human body have been used in some danceperformances to control sounds.

[0342] The invention provides a generalization of this for synergisticuse in conjunction with others of its aspects.

[0343] Sensors may be attached to the human body by means of clothing,jewelry, straps, adhesive pads, etc. These sensors can be of a varietyof types: position, motion, optical, skin resistance, muscle activity,etc. and may be used to capture body position, posture, activity,environment, etc. and convert these into control signals used to controlsound, lighting, and special effects. Sequences of control signals canalso be interpreted as gestures by recognition systems which in turn canbe used to generate yet other control signals. Interfaces to thesensors, taken collectively as an instrument entity 100, to one or moresignal routing, processing and synthesis entities 120, may be done bymeans of radio, wireless optical, fiber optic cable, electrical cable,or combinations or sequences of these.

[0344] Although the sensors described here taken as an instrument entity100 may be used in isolation, there is particular synergistic value inusing these in conjunction with other instrument entities in aperformance or recording situation. For example, a particular bodymotion or gesture (such as raising an arm, swinging a hand, jumping,etc.) may have significant artistic value at a critical moment but notbe captured by another instrument entity. As another example, inrecording sketches during a composition phase, particular body motionsor gestures can be used to call attention to specific aspects of thesketch for future review.

[0345] 2.1.11 Stage Environment and Macro-Environment Sensors

[0346] Sensors other than optical can be distributed on a stage and/oron component installations on the stage (for example staircases, risers,scaffolds, sculptures, props, etc.). Sensors can also be used to measurelarger environments ranging from audience activity to outdoormeteorology. The sensors can include proximity, position, motion,weight, temperature, humidity, etc. and can be used to create controlsignals. As a result, these arrangements can be formalized into aninstrument entity 100.

[0347] Examples of such usage include human proximity and/or interactionwith props or sculptures, tracking of artificial fog cloud migrationacross a stage, detecting the location of performers on staircases orrisers, detecting audience motion activity, characterizing room-internaland room-external meteorology (such as wind speed, wind direction,rainfall, wind and/or rainfall noise, etc.) to bring it into an aspectof the performance.

[0348] 2.2 Vibrating-Element Instrument Elements and Subsystems

[0349] 2.2.1 Single-channel audio signal handling

[0350] The invention provides for the inclusion of traditional group (or“composite”) audio signals such as a group pickup serving all strings ona traditional electric guitar. These can be treated as a peer to any ofthe multi-channel audio signals or of special significance because ofits timbre, functionality, or traditional use. As will be illustrated inthe discussion of layered signal processing, such a signal can beprocessed so as to create the subtle or dominate backdrop against whichprocessed multi-channel signals are superimposed. In some situations,multi-channel signals on the instrument may be combined to create asingle channel audio output, as in the case where individual piezobridge pickups are only one of a plurality of multi-channel signalsources on an instrument; simple full or partial mix-downs may beprovided for use when such multichannel sources are not featured in amulti-channel manner so as to conserve channel usage on the generalizedinterface 110. This can be particularly valuable in complex instrumentswith many arrays of vibrating elements such as those in FIGS. 4-5 andmany others to be discussed.

[0351] 2.2.2 Multi-channel audio signal handling

[0352] The use of various types of musically-oriented signal processingwith electronic stringed instruments has been common in popular musicalmost as long as there have been electronic stringed instruments.Typically a single pickup is used to capture audio signals from allvibrating elements on the instrument (although there may be a pluralityof such group pickups on a given instrument so as to obtain differentselections of timbre).

[0353] The invention provides for the use of multi-channel electrictransducer arrangements, by which each vibrating element (string, tyne,membrane, etc.) of an electronic instrument with multiple vibratingelements is provided with an independent isolated electrical output, anddedicated signal processing can be applied to the signal of eachvibrating element or incomplete combinations thereof, to achievesignificantly important musical functions -- all done in a way where thesame interfaces, multi-channel signal routing and processing, andinternal instrument electronics can be reused across a variety ofinstruments.

[0354] Multi-channel vibrating element pickup arrangements, by whicheach vibrating element (string, tyne, membrane, etc.) of an electronicinstrument with multiple vibrating elements is provided with anindependent isolated electrical output, have been commercially availablebut in largely hidden forms, most commonly used in synthesizerinterfaces for guitars. Beyond such synthesizer interfaces, and therecent Roland VG-1 product discussed later, the usage of suchmulti-channel vibrating element pickups has been limited to roles involume equalization and imaging in a stereo sound field on only a veryfew electric guitars models. Such musically-oriented signal processingis only known to have been applied to the summed mixture of allvibrating elements of the instrument, not for individual or subgroups ofthe vibrating elements of the instrument.

[0355] Conventional signal processing can be used on each vibratingelement signal to create “generalized pedal steel guitars” (augmentingor replacing mechanical pedal tuning changers), instantly retunableguitars (augmenting or replacing mechanical tuning changers such as theHip-shot “Trilogy”), multi-modal Indian sitars (where drone andsympathetic strings can be electronically retuned while playing,allowing a more robust mix between Eastern and Western tonality inmusical form), spatially animated instruments where individual vibratingelement sounds are location modulated within a stereophonic or otherspatial sound field, and mixed timbre instruments where different signalprocessing methods are applied to each string.

[0356] Standard pickup elements available to implement individualpickups for each vibrating element include piezo contact elements,installed on a bridge acoustically isolated from other vibratingelements, and non-contacting coil-based electromagnetic pickup elements.Optical pickup products have also been devised, and a coil-lessHall-effect pickup method has been taught as U.S. Pat. No. 4,182,213.Both optical and Hall-effect methods do not involve contact with thevibrating element. FIG. 20 shows electromagnetic, Hall-effect, piezo,and optical pickup methods for deriving separate audio signals for eachvibrating element of a multiple vibrating element instrument entity.Electromagnetic coils and Hall effect elements require the stringmaterial to be ferromagnetic while piezo and optical methods do not.

[0357] It is noted that a pickup localized for individual vibratingelement must by its nature have small geometry. For the pickuptechnologies not involving contact with the string (e.g.,electromagnetic coils, Hall effect, and optical) multiple small pickupscan be aligned along a vibrating element's length; the resultingmultichannel signal may be handled with multichannel signal processing,selected by a switch, selectively mixed/morphed, etc. to obtain a rangeof tones. In one implementation the selection, mixing, morphing of thepickup signals, and hence the resulting output tone, may be operated bycontrol signals.

[0358] It is noted that excessive magnetic fields from a large number ofmagnetic pickups may make a low-mass vibrating element such as a thinstring vibration go inharmonic. Although this should be a designconsideration with a number of pickups, it can also be used to producespecial effects. The invention thus provides that one or moreelectromagnetic coils, which may or may not otherwise double as pickups,be used to issue localized DC magnetic fields of varying intensity forinducing inharmonic effects on one or more selected strings, mostadvantageously under control signal control. The coils may create the DCmagnetic fields themselves or instead cause a permanent magnet to varyits distance to the vibrating element via solenoid structures.

[0359] The sloped bridges of sitars and other twanging/buzzing stringedIndian instruments have not to date lent themselves to individual piezobridge structures. This is not impossible; the invention provides forindividual miniature sloped bridges, one for each string, to be embeddedwith its own piezo pickup element. Such bridges can also be used withnon-string vibrating elements, such as bars and tynes, to create newtypes of sounds. This method can also be adapted to the very gradual andsofter sloped body contact of certain African harps whose strings buzzagainst a typically animal fur-covered harp body. Alternatively, FIG. 21shows how an off-bridge buzz-plate, such as those provided by Biax tosimulate a fretless based sound with a conventional fretted bass, may becombined with a piezo bridge sensor in replacement of a gradientbuzz-bridge so as to permit the use of non ferromagnetic strings.

[0360] 2.2.3 Vibrating Element Excitation

[0361] The use of “controlled (acoustic) feedback” with electronicstringed instruments has been in common use in popular music since atleast the 1960's. It has been possible to replace the acousticexcitation of string resonance with electromagnetic excitation (asembodied by the Heet Sound E-bow) for some time, but only for one stringat a time and via hand-held mechanically operated apparatus. Thepractice of electromagnetic excitation in nonstringed musicalinstruments with vibrating elements is not currently known.

[0362] The invention presents a system using electromagnetic excitationof the vibrating elements of an electronic instrument to producecontrolled feedback relationships with signal processing control of thefeedback characteristics, typically hands-free as desired, with eitherstandard parts (for inexpensive mass manufacture and retrofit) or morespecialized parts (to provide additional features).

[0363]FIG. 22 shows the basic idea of controlled feedback as used inrecent contemporary music circa 1960. A vibrating element 2201 withinthe instrument is coupled (by electromagnetic, optical, or mechanicalmeans) to an electrical transducer 2202 (electromagnetic, optical,Hall-effect, piezo, etc.) which converts the vibration to an electricalsignal 2203. The electrical signal 2203 is applied to a power amplifier2204 which drives a loudspeaker 2205 which is acoustically coupled 2206(by means of air, mounting apparatus, etc.) to the vibrating element2201. This creates a feedback arrangement allowing vibrations ofspecific frequencies to resonate within the resulting closed-loopsystem. By exciting or damping the vibrations of the vibrating element2201, changing the characteristics of the frequency and/or phaseresponse of the speaker's power amplifier 2204, and/or changing thecharacteristics of the means of acoustic coupling 2206 (as in changingthe distance between the vibrating element 2201 and speaker 2205), the“controlled feedback” methods used in popular music are obtained. Note,however, that the required acoustic coupling characteristics areaffected by factors such as volume level (typically this must berelatively high), speaker/room geometry, room acoustics, and otherdifficult to control factors that can often be unpredictableliabilities.

[0364] The invention provides for an approach to replacing the acousticexcitation component of this process with electromagnetic excitation.FIG. 23 shows an example implementation of simple approach for replacingacoustic excitation of a vibrating element with electromagneticexcitation. In this case the vibrating element 2201 must beferromagnetic, although the transducer need not use ferromagnetic meansitself. Here, rather, the acoustic coupling 2206 is replaced byelectromagnetic coupling 2209, produced by an electromagnetic coil 2208with an internal magnet or other magnetic bias that replaces the speaker2205. Without this nominal magnetic field, the string will be excitedwith a full-wave rectification as the ferromagnetic string is drawn tothe coil regardless of the direction of current flow. In thisconfiguration, the power amplifier must now match the coil's electricaldrive requirements which can differ from the speaker's, hence thespeaker's power amplifier 2204 is replaced by the coil's power amplifier2207. A signal output 2210 for subsequent amplification and signalprocessing can be taken off at the transducer. (In particular, thisoutput arrangement 2210 differs from the Heet Sound E-bow where no suchoutput is provided or relevant for that product). The result is a systemthat provides a comparable arrangement to that of FIG. 22 without therequirements of high volume level, speaker/room geometry, roomacoustics, and other liabilities of acoustic coupling.

[0365] It is also noted that as piezo elements both convert vibrationsinto alternating current signals and, reciprocally, convert alternatingcurrent signals into mechanical vibrations, a piezo group element bridgepickup can be used, in lieu of a coil, either as the audio signal pickupor as a mechanical drive exciting element. Further, the signal pickupcan also be optical or Hall effect. If both the signal and driveelements are electromagnetic (coils or Hall for signal pickup, coil fordrive) undesirable magnetic coupling, not unlike that of an electrictransformer, can occur. This effect may be minimized if said signal anddrive elements are sufficently separated and/or shielded or otherwiselocalized (for example, with a two-coil/opposite-magnet arrangement.

[0366]FIG. 24 shows various combinations of piezo and electromagneticvibrating element pickups and exciter drivers for separatelycontrollable excitation of each vibrating element. In one arrangement,the string 2400 suspended over the bridge 2401 is electromagneticallycoupled to two electromagnetic coil pairs 2411 and 2412. Each coil pairis in a standard “humbucking” arrangement with complementary magnet poledirections 2413 a, 2413 b and complementary winding directions 2414 a,2414 b so as to significantly localize the magnetic coupling regionabout the coil. Here either coil may be used as the signal source or asthe driver. The source coil can also be replaced with an optical orHall-effect pickup. In a piezo-based arrangement, the string 2400 is incontact with a piezo pickup element 2421 on the bridge 2401, and thestring 2400 is magnetically coupled with an electromagnetic coil pair2422. Here either the coil pair or the piezo may serve as the driver andthe other serve as the signal source. In cases where the piezo elementacts as the driver, the electromagnetic signal source element may bereplaced with a Hall effect or optical technology pickup.

[0367] It is noted that the invention provides for the above discussionsto apply equivalently should the signal source and driver elements servean individual vibrating element or a group of vibrating elements. Theinvention also provides for the case where either the signal source ordriver is a single element unit while the other is a group element unit;such configurations are easily supported by the signal routing,processing, and synthesis entity 120 (referring to FIG. 1). Inparticular the invention provides for the case where a multiplevibrating element instrument has at least one of the following:individual source and individual excitation, group source and groupexcitation, individual source and group excitation, and/or group sourceand individual excitation.

[0368] Since the driving element (coil or bridge piezo) may be mountedin permanent relation to the vibrating element, it is possible toreplace conventional means of altering the acoustic coupling withelectronic signal processing means 2211. FIG. 25 shows adding signalprocessing for spectral and amplitude control of electromagneticexcitation. For example, fixed or adaptive equalizers can be used toalter the frequency and phase response of thesignal/vibration/transducer loop, permitting additional control overwhich vibrational harmonic(s) are emphasized in the feedback.Attenuation can be used to vary the degree of feedback. Delay can beused to alter the attack characteristics of the resonance behavior.Dynamic compressors and expanders can be used to vary the ease anddynamics of the resonance behavior. Many interesting special effects arepossible, such as using pitch-shifters alone or in combination withdelays to transfer energy between vibrational modes. In the case of adrive coil, a pitch shifter or octave divider (such as an inexpensivetoggle flip-flop) may be used to create a drive signal that is an octavelower than the string signal and thus eliminate the need for a magnet inthe drive coil. The invention provides for any driver electronics and/orsignal processing to be any of the following: internal to theinstrument, mounted on the outside of the instrument as an add-onmodule, remotely located off the instrument (particularly in the signalrouting, processing, and synthesis entity 100), or any combinationthereof.

[0369] In most electronic instruments, a single pickup serves many ifnot all the featured vibrating elements. The invention provides for theapproaches discussed thus to also be applied to such instruments usingconventional components. FIG. 26 shows an example arrangement involvingmultiple vibrating elements served with a group pickup and which arealso subjected to common electromagnetic excitation using conventionalguitar pickup components. (In the example of FIG. 26 it is understoodthat signal processing 2211 may be introduced or omitted from thefeedback loop as appropriate or desired and that a group piezo bridgemay, in some constructions, serve as a driving element in place of thepickup coil.). A plurality of vibrating elements 2201.1-2201.n share acommon group pickup transducer 2202 and a common electromagnetic coil2208. This arrangement is very simple to implement and very usefulmusically for traditional electric guitarists. A simple prototype can bemade using an electric guitar with two humbucking-pickups (such as aGibson ES-335). The rear pickup can be used as the transducer 2202 andthe front pickup as the coil 2208 (the humbucking pickups assist indecoupling the coils, decreasing a parasitic “transformer” effect).Almost any power amplifier of sufficiently high enough current orvoltage drive, (for example, even a Fender Bassman tube guitaramplifier) can be used as the coil's power amplifier 2207, directlydriving the coil (despite the impedance mismatch) from the amplifierspeaker output connector.

[0370] 3 Example Electronic Controller Instruments

[0371] 3.1 Touch-Pad Array

[0372] Touch pad instrument elements, such as null/contact types andpressure-sensor array types described earlier, can be used in isolationor arrays to create electronic controller instruments. The touch-pad(s)may be advantageously supplemented with panel controls such as pushbuttons, sliders, knobs as well as impact sensors forvelocity-controlled triggering of percussion or pitched note events. Inthe case of null/contact touch-pads, impact and/or pressure-sensors canbe added to the back of the pad and the pad suspended in such a way thatit can be used as an electronic drum head. If one or more of thetouch-pads is transparent (as in the case of a null/contact touch screenoverlay) one or more video, graphics, or alphanumeric displays mayplaced under a given pad or group of pads.

[0373]FIG. 27 illustrates examples of single, double, and quadrupletouch-pad instruments with pads of various sizes and supplementalinstrument elements. A single touch-pad could serve as the centralelement of such an instrument, potentially supplemented with panelcontrols such as push buttons, sliders, knobs as well as impact sensors.In FIG. 27, a transparent pad superimposed over a video, graphics, orone or more alphanumeric displays is assumed, and specifically shown isa case of underlay graphics cues being displayed for the player. Twolarge sensors can be put side by side to emulate theleft-hand/right-hand layout of many hand drum arrangements such astabla/baya, congas, etc. This is particularly suitable forpressure-sensor array touch-pad elements where a larger pad-area (forexample 8 to 12 inches square) could be advantageous for detailedcontrol. Because of the extensive capabilities of either type oftouch-pad element provided for in the invention, this arrangement is byno means limited to percussion applications but rather easily serves asa far more general purpose left-hand/right-hand multi-parametercontroller. In variants of this arrangement that are intendedspecifically for tabla/baya emulation, the relative size of the two padsand angle of placement with respect to the floor can be arrangement tomatch that expected by an experienced tabla player. Instrumentsinvolving arrays of larger numbers of touch-pads can also be valuable.Here it may be advantageous to make the pads smaller so that the fingersof a single hand can touch two or more pads simultaneously.

[0374] 3.2 Foot Controllers

[0375] With the extensive real-time control capabilities provided for inthe invention, foot controllers can be especially valuable. They canselect preset configurations at various points in a control hierarchy,issue notes or chords, control timbre, alter lighting, invoke specialeffects, etc. In general a commercially available floor controllertypically includes a plurality of momentary action foot-switches, andvarious visual status indicators such as LEDs over momentary actionfoot-switches and a master status (and programming) display. Many suchproducts also include provisions for rocker foot pedals to controlcontinuous parameters, either via external connection (as with theDigitech PMC-10 and Digital Music “Ground Control” products) orinternally (as with the ART X-15 product). With the exception of theDigitech PMC-10, the control assignment and organization capabilities ofthese controller products have historically been quite limited, and asall the products seem aimed largely at issuing MIDI program changecommands, the number of foot-switches has been small. Further, therocker foot pedals control only one parameter at a time.

[0376] The invention provides for extensive elaboration over theseproducts by supporting any of multi-dimensional rocker pedals, arbitrarycontrol signal assignment, control signal assignment organized byselectable pages, separate alphanumeric function display for each footcontrol (switches and pedals), pause operations, and real-time eventplay-back capabilities.

[0377] The traditional way to control volume on an electronic keyboardinstrument is by a means of a rocking floor-level foot-pedal. Morerecently such pedals have been used to generate continuous-range controlsignals such as MIDI messages, though allowing the control of only onecontinuous-range parameter at a time. Many years ago a number of“volume/tone” foot pedal products were available, though none appearavailable at this writing. These products offered a rocker capabilitydevoted to controlling instrument volume supplemented with a left-righttwist capability devoted to the control of instrument tone. Sucharrangements may be used to double the number of foot controllableparameters that can be controlled in roughly the same physical layoutarea together with the bonus of allowing a foot to control twocontinuous-range parameters at once.

[0378]FIG. 28 illustrates some enhanced foot-pedal arrangements whichpermit simultaneous single-foot adjustment of a plurality of continuousrange parameters for use with floor controllers. The use of rockingfoot-pedals to control two continuous-range parameters at once may beenhanced by using one or more side-mounted spring-levers. Side-mountedmomentary-action switches have been used on rocker foot-pedals for modecontrol (products by Ernie Ball and Soloton), but side-mountedspring-levers are particularly advantageous for continuous rangeparameters that have a specific nominal value. For example, these can beused in conjunction with pitch-shifters to modulate pitch as do the footand knee levers of a pedal steel guitar, or in complementary pairs toemulate the action of a synthesizer modulation wheel or an electricguitar vibrato “whammy bar.” In FIG. 28, one or more side-mountedspring-levers 2803, 2804 may mount on either the base 2801 or rockerplate 2802 of a foot-pedal. A spring lever may directly operate a slide,geared, pulleyed, etc., potentiometer, an optical sensor, magneticsensor, pressure-sensor, etc. to produce an electrical signal. If twoside-mounted spring-levers 2803, 2804 are positioned on opposite sidesof the pedal, two mutually-exclusive parameter adjustments can berealized (as found in pedal steel knee-lever pairs and in the action ofa synthesizer modulation wheel or an electric guitar vibrato whammybar). It is also possible to mount a springed center-return synthesizermodulation wheel 2805 at the far end of the rocker plate if thearrangement and materials used forego breakage in heavy usagesituations.

[0379] Further, it is possible to add a third control continuous-rangeadjustment capability on the rocker pedal by measuring the length-axisrotation of the foot: this could be done by various methods. As oneexample, a two-dimensional “volume/tone” foot pedal with control motionsup-down 2810 a and twisting 2810 b may be modified to permit length-axisrotation of the foot 2810 c and measure it with a potentiometer orsensor. Another method would involve putting at least twopressure-sensors 2813 on the twist plate 2812 of a non-modifiedtwo-dimensional foot pedal 2811 and deriving a control signal fromthese. A third way would be to mount a springed center-returnsynthesizer modulation wheel at the far end of the twist plate if thearrangement and materials used forego breakage in heavy usagesituations. Other methods can be used for multi-dimensional footcontrollers, such as the null/contact touch-pad and pressure-sensorarray touch-pad elements discussed earlier which can be adapted for footoperation.

[0380] The invention provides for arbitrary assignment of controlsignals to specific foot-switches, foot-pedals, and other footcontrollers. As an example, one or more MIDI messages could be assignedto each foot-switch, foot-pedal, or other foot controller as is largelydone in the Digitech PMC-10 and with other functionality as the custommessage construction and hierarchical ganging provided by, for example,the Peavey PC-1600 slider/button controller).

[0381] A particularly valuable additional function would be that ofissuing continuous controller messages that oppositely complement thebasic control signal value: for example. in MIDI messages where“Continuous Controller” control values lie in the range 0 to 127, if acontinuous foot-pedal position causes a first control signal to beissued with value of “x”, it is also possible to enable the subsequenttransmission of a second separate control signal to be issuedessentially simultaneously with a value determined by the algebraicrelation “127-x”; such complementary signals may be used for manypurposes, for example prorating an audio mix between two sources,prorating modulation indices among two synthesizer voices, etc.

[0382] Stored program memory may be used to retain these assignments. Inthis situation it is advantageous to allow for multiple stored programselections to be recalled, thus allowing for multiple assignment setsfor each foot-switch, foot-pedal, etc. Each assignment set could bethought of as a “page.” Pages could be copied as a whole and edited.These capabilities would be similar to those of the Digitech PMC-10 andPeavey PC-1600 products. However, because of the number of controllerassignments and the diversity of possibilities it is desirable to addphysically adjacent to each foot controller an alpha-numeric displayindicating the current assignment and status of that controller: inparticular, for each given selected page, each controller display mayshow one or more of the currently assigned function, the currentvalue(s) transmitted or last-transmitted, any additional identifyinginformation such as short-hand names or relationships with othercontrollers, etc. LEDs may be provided for quick reference as to whichfoot-switch and which continuous foot controller (pedal, touchpad, etc.)were last operated; as an enhancement these LEDs could be bi-color andof the two LEDs lit at a given instant (one for last foot-switch used,the other for continuous controller used), one color (i.e., green) couldbe used to indicate to overall last operation while a second color(i.e., red) would be used to illuminate the remaining lit LED.

[0383] The operation of a foot-switch may be assigned, under storedprogram control, to issue one or more simultaneous control signals, orshort burst of contiguously-sequential control signals such as a groupof MIDI messages. These control signal events may occur on thedepression of the foot-switch, its release, or both. The foot switch mayalso be configured to operate in a toggle mode using a divide-by-twocounter and messages can be issued on each toggle transition. Theseuseful features can be found on, for example the Digitech PMC-10, but anumber of useful enhancements are provided for by the invention. Oneenhancement would be to allow any specific pedal to independentlyoperate in a generalization of toggle mode to permit a round robinselection of 3 or more states (for example “off,” “slow,” “medium,”“fast”). Another enhancement is to allow a more complicated statetransition map involving a group of foot-switches. Yet anotherenhancement is to permit timed events to be issued. The simplest ofthese would be timed pause operations between control signal events,while a more enhanced implementation would permit real-time controlevent playback capabilities to be assigned to a foot-switch. Suchreal-time event sequences could include not only note sequences but alsotrajectories of continuous parameters (for example, exponentialtransients or linear ramps). Further, the invention provides for theissuance of the same selection of possible control signal options uponincoming or outgoing page-change events during a stored memory pagechange.

[0384] Finally, larger foot controller assemblies with appropriateorganizational and ergonomic layout are advantageously provided for bythe invention. Among the factors here are overall ergonomic operation,putting some foot controlled elements closer to the user for fast orintimate use with others farther away for background or occasional use,and an overall physical and operational organizational hierarchy. Inimplementing such hierarchies, each full stored program page can involveone or more sub-pages which also be used as a part of other full storedprogram page. Although such a sub-page can in general be assigned to anyfoot operated control element, it typically would be useful to confineeach sub-page to a pre-defined reusable geometric region in the overallfoot controller layout. Further, the invention provides for sub-pages tobe changes within an active full page.

[0385]FIG. 29 shows some example layouts involving 2 geometric regionsfor a moderate number of foot operated controllers 2920 and 4 geometricregions for a larger number of foot operated controllers 2940. Thesmaller arrangement 2920 features a rocker pedal with two side-mountspring controllers 2903 and two rock/twist pedals 2900 as well as twogeometric regions—one proximate 2921, another remote 2922—offoot-switches 2905. Each foot-switch and pedal is provided with analphanumeric display 2906 and a last-operation indicating-LED 2907. Thelarger arrangement 2940 features an advantageous layout of two proximategeometric regions 2941, 2942 of foot-switches, two increasingly remotegeometric regions 2943, 2944 of foot-switches, one proximate rock/twistpedal 2900, remote pedals of rocker only 2901, single side spring lever2902, and double side spring lever 2903, as well as a foot or toeoperated touch-pad 2904. Each foot operated controller is also providedwith an alphanumeric display 2906 and a last-operation indicating-LED2907.

[0386] The layout used in the larger unit permits logical association ofgroups of switches and pedals in a wide variety of contexts. In eitherthe smaller or larger arrangement, the more remote controllers can beput on progressively higher risers to create a staircase layout. Thesearrangements permit for an arbitrary logical hierarchy of page andsub-page recall control and arbitrary assignment of which buttons may beused to do this. In some cases it may be desirable to have an additionalsummary display showing the selected page and sub-page status in onelocation at a glance.

[0387] 3.3 Multi-Tier Proximate/Superimposed Keyboards

[0388] The proximate and superimposed keyboard elements describedearlier can be combined to create a powerful enhanced keyboardcontroller. In an example implementation, an arrangement of threeproximate keyboards such as shown in FIG. 3 may be brought together in acommon unit. This unit may also advantageously include one or more ofany of sliders, knobs buttons, joysticks, touch-pads, strum-pads, impactsensors, etc. Further, it is noted that any of the keyboards here may beeither of a standard variety or any of the more advanced keyboardsdescribed later (miniature, superimposed, multi-parameter keys,pressure-sensor array, etc.). It is also noted that this technique maybe applied to other types of keyboards with applicable types of keygeometry.

[0389] 3.4 One-Hand Enhanced-Drum-Roll Controllers

[0390] The invention provides for one-handed methods of performingdrum-rolls with some advanced capabilities. The basis of the methodinvolves the proximate location of two electronic impact sensors and/ortouch pads oriented to be facing each other, but the method can also beused with acoustic drums. The arrangement can be small in scale, i.e.,played with a single finger, or larger to be played with hands, beaters,mallets, or sticks. FIG. 30 shows an example large-scale arrangement oftwo impact sensors and/or touch pads for executing one-handed drum-rollsand deriving large amounts of control information. The figureillustrates a larger-scale arrangement of two impact sensors and/ortouch pads 3000 a, 3000 b supported in the method's configuration by,for example, supporting beams 3003, 3004 connecting to a commonsuspending clamp 3005 on an instrument-stand beam 3006 on one side andjoints 3007, 3008 to the sensors and/or pads 3000 a, 3000 b on the otherside; though, clearly, other mounting arrangements are possible. Thesensors and/or pads 3000 a, 3000 b are separated from one another by adistance that permits a beater 3010, mallet, or stick to be held in onehand at the far end 3011 and rapidly rocked back and forth between thetwo sensors and/or pads so that the beater head 3012 impacts the sensorsand/or pads. The beater may also be held at its center of mass orgeometry 3013 and vibrated so that both the beater head 3012 and end tip3014 of the far end impacts the sensors and/or pads; in this techniquethe player may orient the beater motion so as to simultaneously impactone impact sensor and/or pad with the beater head 3012 and impact theother impact sensor and/or pad with the beater end tip. In this playingtechnique it is advantageous to have provided for some regionaldifferentiation of the impact sensors and/or pads; null/contact pads,for example can do this. Another arrangement is that of two impactsensors, one for the center area 3001 of an impact pad 3000 and theother for the outer rim area 3002. With the ability to differentiateregions of impact, and even non-impact applied contact regions andpressure, the portion contacted by the end tip and head can bedifferentiated. Further enhancement can be obtained by using a beaterendowed with sensors; these can provide contact localizationinformation, as well as hand grip information, which may be usedindependently or in correlation with the information generated by thepads 3000 a, 3000 b. The resulting arrangement allows a performer withone hand to do a wide range of percussion and other control actions,leaving the other hand free for playing another instrument entity orexpressing visual gestures during performance.

[0391] In a smaller scale implementation, one or more fingers can beused in place of a beater. This arrangement can be treated as aninstrument element in itself to be used as part of other instrumententities.

[0392] Regardless of scale, it is noted that two such arrangements canbe colinearly co-located but in 90-degree rotational offset. Thiscreates a rectangular cavity for beats, fingers, etc. to be inserted andvibrated, and additional degrees of control. This can be generalizedinto arbitrary polygonal cross-sections (triangles, pentagons, hexagons,etc.).

[0393] 3.5 Video hand position and gesture

[0394] A camera with appropriate real-time image processing may be usedsimultaneously or mutually exclusively as an instrument element as wellas a video feed source for recording or performance. As such the cameramay be treated as an instrument element mounted on an instrument entity,but can also be used as a self-contained instrument entity. For example,a camera could be aimed upwards and surrounded by illuminating lights. Aperformer can activate and control this instrument entity by putting ahand over the camera and executing various positions and gesturesrecognized by the image processing capabilities.

[0395] 3.6 Video Stage Tracker

[0396] A camera may also be used to transform visual informationobserved from a stage into control signals. The relevant imageprocessing and recognition capabilities may advantageously includeidentifying and tracking performer location and motions.

[0397] 4 Example Adapted Instruments

[0398] This section discusses example manners and methods the inventionprovides by which a number of traditional vibrating element instrumentscan be enhanced by incorporating various synergistic combinations oftraditional components and the invention's instrument elements.

[0399] 4.1 Autoharp

[0400] A traditional autoharp incorporates a plurality of strings, tunedto selected notes in a chromatic scale, which are selectively damped bymechanical damping bars with cut-outs in the damping material that allowonly selected strings to sound. A player selects and activates a damperbar associated with a chord and strums a portion or all of the strings,and only the undamped strings, namely those associated with the voicingof the chord, sound. Although at times considered a lower folk orbeginning instructional instrument, the basic arrangement of theautoharp can give rise to a powerfully flexible instrument.

[0401] In its simplest provision, the invention provides for an autoharpto be supplemented with sliders, switches and buttons for issuingcontrol signals. In particular, a select group of buttons or contactscan be operated by, or in conjunction with, the mechanical damper bars.This group of buttons or contacts may be used to control at least one ofthe following: issued note control signals for sound, lighting, and/orspecial effects, note assignments to one or more strum-pads, and/or theamplification of individual strings. The individual strings of theautoharp may have one or more of the following: a common pickup for theentire group of strings, a plurality of smaller pickups associated withsub-groups of strings, or a full plurality of individual pickups foreach string. The pickups may be any of electromagnetic, piezo, optical,etc. in their operation. In cases where a plurality of pickups isemployed, signals from groups of strings or individual strings may behandled by multi-channel signal processing as described later (forexample, treating the strings with differing degrees of equalization,chorus, reverb, pitch shift, dynamic filter sweeps, etc., and/orproviding separate noise gates, compression, limiting, amplitudecontrol, etc.). In cases where each string has its own pickup, theplucking of a particular string may further be used to trigger asynthesizer note, lighting, or special effect event, potentially usingthe amplitude of the pluck to set note velocity and potentially trackingthe ongoing string amplitude and even harmonic structure variations asprovided for in the invention and described later. Strum-pads may beprovide for use in conjunction with strumming the strings or inconjunction with operating the mechanical chord dampers. Controls may beprovided for stored program recall of control signal assignments,strum-pad voicings, etc. as well as operational features such muting orsustaining of strum-pad notes, whether notes issued at the pressing of achord damper bar are released when the damper bar is released or insteadonly when a new bar is activated, etc. These control features may alsobe controlled remotely, for example, with a foot controller, and/orimplemented remotely in a separate signal routing, processing, andsynthesis entity 120.

[0402]FIG. 31 shows an example of an enhanced autoharp implementation asprovided for in the invention. A conventional autoharp 3100 with theusual arrangement of strings 3101 and damper-bar activating chordbuttons 3102 may be fitted with a long strum-pad 3103 adjacent to thetraditional strumming area, one or more shorter strum-pads 3104 a, 3104b near the chord button area, a plurality of slider controls 3105 andcontrol switches 3106, and control buttons for stored program recall3107 a, operational mode control 3107 b, or other features 3107 c.

[0403] As another part to the invention, the mechanical chord damper bararrangement may be advantageously replaced with a 12-note keyboard orsimilar arrangement for selecting which chromatic notes are allowed tosound. String damping control may be done mechanically although thisrequires damper bars to normally damp selected strings and let thosewanted strings sound only when a key or button is depressed (rather thandamping only unwanted strings when a key or button is depressed). Inthis way more arbitrary chords can be selected, chords can bedynamically changed at a resolution down to one pitch at a time, etc.Alternatively, if a separate pickup can be provided for each string,mechanical string sounding control may be replaced with electronicamplitude control. In the simplest form, all strings of various octavesof the same note are gated on and off by the depression of the key onthe keyboard associated with that note. If the key depression-depth ortotal pressure on the key is used as a volume control, the relativevolume of all octaves of each pitch can be controlled independently fromthat of other pitches. If the key further has two-dimensional touchsensing, as with a null/contact touch-pad on each key, balance betweenvarious—typically four—octaves can be continuously varied (for exampleleft/right controls the balance between octaves 1 and 2 and in/outcontrols the balance between octaves 3 and 4, thus allowing arbitrarybalance choices of the four octaves). The multi-parameter key control ofthe amplitude and mix of each sounded note is of particular value whilethe string sounds after the note is initiated. The keyboard,multi-parameter or not, can also be used to control similar aspects ofnote assignments and amplitudes of synthesizer notes initiated with eachstrum-pad.

[0404]FIG. 32 shows how the autoharp arrangement of FIG. 30 can beadjusted to replace the chord button array 3102 and associatedstrum-pads 3104 a, 3104 b with a keyboard 3202 and one or morestrum-pads 3204 positioned over the keyboard.

[0405] 4.2 Harps, Koras, Zithers, Kotos, Mbiras

[0406] The enhancements of Harps, African Koras, Zithers, JapaneseKotos, African Mbiras, and other related instruments with a large arrayof hand-plucked vibrating elements are also provided for as part of theinvention. As with the above autoharps, pickups may be used for allvibrating elements, or, advantageously, sub-groups of elements, or—mostadvantageously—separately for each vibrating element. The pickups may beany of electromagnetic, piezo, optical, etc. in their operation. Theinvention also provides for the instrument to be supplemented withstrum-pads, touch-pads, sliders, switches and buttons for issuingcontrol signals and affecting internal operation and note-event handlingmodes.

[0407] In cases where a plurality of pickups are employed, signals fromgroups of vibrating elements or individual vibrating elements may behandled by multi-channel signal processing as described later (forexample, treating the strings with differing degrees of equalization,chorus, reverb, pitch shift, dynamic filter sweeps, etc., and/orproviding separate noise gates, compression, limiting, amplitudecontrol, etc.). In cases where each vibrating element has its ownpickup, the plucking of a particular vibrating element may further beused to trigger a synthesizer note, lighting, or special effect event,potentially using the amplitude of the pluck to set note velocity andpotentially tracking the ongoing string amplitude and even harmonicstructure variations as provided for in the invention and describedlater. Strum-pads may be provided for use in conjunction with pluckingthe vibrating elements.

[0408] Harps, Koras, Zithers, Mbiras, and other related instruments witha large array of hand-plucked vibrating elements often have onlyselected pitches available; accidentals and extreme octaves typicallyare not represented. Many of these instruments allow for accidentalsduring playing, for example harp tuning levers and Koto string bends,while others, such as the Mbira, do not; in almost all cases extremaloctaves are not supported at all (aside from execution offundamental-muting string “harmonic chiming” to attain high octavepitches). With each vibrating element (or, less flexibly, groups ofvibrating elements) provided a separate pickup and audio channel, pitchshifting can be used to electronically obtain pitches not provided forby the natural form of the instrument as well as large expressive pitchbends that may also not otherwise be possible.

[0409]FIG. 33 shows an example Koto implementation provided for inaccordance with the invention. In general a Koto includes a number ofstrings 3301 (typically 13 to 22 in number), a sounding bridge 3302, anda movable truss bridge 3303 for each string. The Koto 3300 shown in FIG.33 is of the Vietnamese variety, traditionally strung with sixteen metalstrings; in general any traditional Koto (or Chinese Sheng) can beadapted as will be described and may have any traditional number ofsilk, nylon, metal, or other material strings. In this example the Kotohas been fitted with geared string tuners split into two groups 3304 a,3304 b to facilitate radical tuning changes and string replacement;other nominal string tuning mechanisms, including traditional frictionpegs or slip knot systems, may also be used. Each string can be givenits own pickup 3306.1-3306.n, at the bridge 3302; alternatively, or inaddition (should the string be such that electromagnetic, optical, orother non-piezo pickups be applicable) at a different string location inone or more pickup housings 3305. Multi-channel signal handling asdescribed earlier and later on can be used. The Koto can be fitted witha strum-pad 3310 and may also be provided with various additionalsensors and controls as described earlier. Because of the uniquepitch-bending arrangement of the Koto involving varying the tension ofthe string on the non-plucked side of the movable truss bridges 3303, itmay also be advantageous to provide strain gauges on far bridge 3307 orvia, for example, flexible attaching electrical cables on the trussbridges 3303 themselves.

[0410]FIG. 34 shows an example Mbira implementation provided for inaccordance with the invention. Here a traditional Mbira 3400 with threesets of tynes 3401, 3402, 3403 secured by a bridging pressure bar 3404is shown. Here the traditional bar bridge may be replaced withindividually adjustable bridge elements 3406.1-3406.n which may includeseparate piezo pickups for each tyne; alternatively electromagnetic oroptical pickups may be provided under each tyne. The Mbira may also beprovided with one or more strum-pads 3407, 3408 which may be full lengthor partitioned into right and left segments 3407 a, 3407 b, 3408 a, 3408b for use with the thumbs of both hands.

[0411] Any of these instruments may also be provided with vibratingelement excitation employing the methods presented earlier inassociation with FIG. 24. In addition, as the timbre of theseinstruments is typically shaped by the sympathetic vibration ofunplucked strings on the instrument, it may also be advantageous toexcite, as a group or individually, a number of vibrating elements suchas tynes or strings using the methods described earlier; thesesympathetically vibrating electronic elements, which need not be mountedto the instrument, can then produce audio signals that can be used forambient effects. Alternatively, a dedicated group of sympatheticallyvibrating elements can be mounted to the instrument and excitedmechanically rather than electrically. If the sympathetic vibratingelements have individual pickups, some of the elements can beselectively turned off or attenuated so as to thin out or spectrallysculpt the ambient effect.

[0412] 4.3 Single-course Guitars and variations

[0413] One of the most versatile instruments available for the range oftimbre and expression is the electric guitar which is sadly not oftenused seriously in music composition due to its origins and significantrole in popular music. (In fact, at this writing, even toy pianos aretaken more seriously than the electric guitar!) Part of the reason forthe immense range of timbre and expression is the fact that it is one ofthe few instruments where both hands can be in direct contact with thestring. Another important reason is the range of timbres that can resultfrom string pickups followed by a wide degree of signal processingmethods that have been developed and can be applied. Although therecontinue to be developments in basic electric guitar themes, theinvention provides for significant enhancements of the electric guitaras a powerful instrument entity.

[0414] An important first step is the provision of separate audio signalpickups for each string; these may be electromagnetic, piezo, optical,etc. This allows for multi-channel signal processing as will bedescribed later (for example pitch shifting particular strings for bigbass notes, enhanced processing for strings playing solo lines to standout from strings playing background material, etc.). Strings may begiven one or more dedicated or shared pickups at different points alongthe string's length so as to capitalize on the different harmonicstructure and dynamics offered by different pickup locations. Aplurality of pickups dedicated to the same string or same group ofstrings can be selected or mixed, potentially in adjustable phaserelationships, statically and/or varying in time, on the instrumentand/or externally. Further, selected strings may be excited byelectromagnetic, piezo, or other methods to give a continuously soundingbowed effect whose inter-note attack can be controlled by variousfretting techniques. Additional strings arranged to serve as a harpelement, bass notes as on an arch-lute, or for sympathetic vibration mayalso be provided, as may tynes or other vibrating elements used insimilar ways. Strum-pads, sensors, sliders, joysticks, buttons,touch-pads, actuators, etc. may also be added to issue control signalsto any of signal processing, lighting, synthesizer, or special effects.Similarly, video cameras can be used to generate control signals and/orfor video image feeds in performance or recording.

[0415]FIG. 35 shows an example electric guitar implementation inaccordance with the invention based on a Gibson model ES-335 guitar orother instruments of that style. The invention's enhancements shown canbe added on as modules, added collectively, or built-in. The core guitar3500 features six strings 3501.1-3501.6, a bridge 3502 which may provideseparate piezo pickups for each string, an adjacent hexaphonic pickupmodule 3503 which provides separate electromagnetic, optical, or otherpickups for each string, and additional individual string pickup module3404 for the three lower pitched strings 3501.4-3501.6, two sharedpickups of a two-coil humbucking design near the neck 3505 a and nearthe bridge 3505 b, shared pickup volume and tone controls 3506, and ashared pickup selector switch 3507. Individual strings may beelectronically excited by means of any of a hexaphonic electromagneticdrive unit 3508, a hexaphonic pickup used as a drive unit (or designedas a drive unit) built into the forward shared pickup 3505 a, or bymechanical excitation via the piezo elements in the bridge 3502.Individual string excitation, or even string activity in general, can bevisually indicated by an LED (preferably high-brightness) array 3509under the strings. Group string excitation may be realized by whicheverof pickup 3505 a or 3505 b is not in use, an additional module under thestings, or by group mechanical excitation via the piezo elements in thebridge 3502. Individual string pickup gain normalizing adjustments foreach hexaphonic pickup can be made available for screwdriver adjustment3511 a, 3511 b on an add-on box or built-in panel 3512, shown in theFigure as an add-on box with generalized interface connectors 3519 onits back downward side. The instrument may also be provided with anarray of control switches 3513 and sliders 3514, individual stringpickup selector switches 3516, a video camera 3517 aimed at the playingarea but also useful for hand posture and gesture control, and an area3518 for additional controls, touch-pads, string or tyne arrays, etc.

[0416]FIG. 5 shows an example electric guitar implementation inaccordance with the invention based on a Gibson Explorer model guitar;the invention's enhancements shown can be added on as modules, addedcollectively, or built-in. The core guitar 500 includes six strings501.1-501.6, a locking-nut vibrato bridge 502 a with string-tension“whammy bar” 502 b and fine tuners 502 c, a hexaphonic electromagneticor optical pickup module 503, a shared pickup 505 a near the neck andanother 505 b near the bridge, a shared pickup volume control 506,shared pickup selection, mixing, phase, and coil-shunt switches 507. Theinstrument may also be fitted with a hexaphonic electromagneticexcitation module 508 with string drive indicated LEDs 509. Alsoprovided for by the invention are controls for creating control signals:an array of switches 513, knobs 514 a, one or more expression wheels 514b, one or more joysticks 514 c, two side-by-side strum-pads 516 a, 516 bwhich can be operated as one long strum-pad, chord buttons 520 asdescribed for the autoharp but here used without strings and rather inconjunction with the strum-pads, a touch-pad 522, two miniaturekeyboards 521 a, 521 b, a miniature harp/sympathetic string set 524 withgroup or individual pickups partitioned in string triads with separatebridges 523 a, 523 b, 523 c and tuning heads 525, and a plurality ofimpact sensors. Internally the whammy bar may operate a discrete orcontinuous position sensor and the instrument may also include motionsensors (such as accelerometers) or position detectors (radio,ultrasonic, optical, etc.). The generalized interface connector area 519in this instance is shown built into the instrument.

[0417] 4.4 Baroque and 12-string Guitars, Lutes, Tars, Setars, Saz, Oud,Mandolins, Mandolas

[0418] These instruments involve double-strings. In addition to thetechniques and additional instrument elements, each double-string pairmay share an individual pickup, or each string within in a double stringpair may have its own pickup. At this writing the best more for thelatter appears to be piezo pickups at the bridge due to limitations inlocalizing magnetic fields for such close geometries but optical orother methods could be devised. With a separate signal for each stringwithin in a double string pair, either of the strings can be selectivelydisable, pitch-shifted, equalized, etc. along with other capabilitiessuch as adjustable balance, stereo spatial output, opposing locationmodulation trajectories, etc. Further, as a combined double-stringsignal would confuse audio-to-note information conversions, separationof the string signals for a given string pair enables control extractionsuch as conversion to MIDI note functions.

[0419]FIG. 36 shows an example of an adapted European arch-lute with amix of single strings and double string pairs. The European arch-lute3600 with various ranges of extended fretted and unfretted bass stringsis a noble instrument which with amplification, signal processing, andmulti-channel signal handling can contribute greatly to electronicmusic.

[0420]FIG. 36 shows a close-up of the strings at the bridge 3620 for afourteen string version of the lute with six extended-length unfrettedbass strings: the two highest-pitched melody strings 3601, 3602 and thesix extended-length unfretted bass strings 3609-3614 are single whilethe remaining courses 3603-3608 are double-strung. Of the double-strungstrings the higher pitched ones are typically in unions while the lowerpitched ones are paired in octaves. It is understood than many otherstring configurations, varying in how many strings and which are singleor doubled, exist and can be adapted as described herein. In fact noelectric lutes are known at this time, thus it is novel simply toinclude a group piezo pickup in the bridge 3620 for all the strings soas to bring the instrument into the world of amplification and signalprocessing. A next level refinement would be to provide separate grouppickups for the extended bass strings and the rest of the strings sothat special equalization or other effects can be applied to the bassnotes in a manner differing from the other strings. A next level ofrefinement would be for the bridge 3620 to provide an individual pickupfor each individual single string course or string-pair, while a finalrefinement provides a separate piezo pickup for every individual stringon the instrument.

[0421] It is understood that various controls, strum-pads, etc. may alsobe added in the manner described for previous instrument examples. It isalso understood that the methods described also apply to otherdouble-strung instruments such as 12-string guitars, Saz, Oud, Mandolin,etc. Many of these instruments may also benefit from an additional setof unfretted bass strings as incorporated in the traditional Europeanarch-lute.

[0422] 4.5 Pedal Steel Guitars

[0423] The pedal steel guitar is a remarkable instrument in that thepitches of individual strings are changed as a group by a hand-heldmetal slide and relatively within the group by mechanical bridgearrangement, usually called a “changer,” which changes the tension onone or more selected strings in response to the action of a givenfoot-pedal or knee lever. The basic sound of the steel guitar is veryattractive and it is possible to tastefully play Bach chorales and hymnson the instrument. Years of incremental development have lead tospecific standard pedal and knee lever configurations that are widelyaccepted. Variations are sometimes difficult to implement because ofmechanical limitations to provided adjustments.

[0424] Because of the commitment involved in mechanically establishingan alternate pedal and lever configuration, immense experience and/or acomputer-aided design tool may be required to make valuableaccomplishments. By providing a separate pickup for each string,retuning can be done electronically, supplementing or replacing thetraditional mechanical mechanisms. As with other adaptations ofinstruments described thus far, each string can also be processedseparately or in groups as desired, allowing for mixes of timbres, andaudio-to-control signal extractions can be used to control synthesizers,signal processing, lighting, and special effects. Further, the nearlyfixed position of the picking hand and the freedom of some fingers inadapted playing techniques allow usage of miniature keyboards andstrum-pads in the picking area as will as use of the wrist to controlparameters. Information from the mechanical or electronics pedals andlevers and the steel bar position can be used to control the pitchesassigned to a strum-pad. The bar itself can have a control areabuilt-in, detecting applied pressure, for example.

[0425]FIG. 37 shows an example pedal steel guitar adaptation as providedfor by the invention. A traditional pedal steel instrument arrangement3700 is used here. The bar 3702 may or may not be provided with internalsensors or controls. The position of the bar 3702 over the instrument3700 may be sensed by various means (changes in round-trip stringresistance, capacitive sensing, etc.) within the instrument 3700. Theinstrument may include a traditional mechanical changer 3740 withlinkages to pedals 3741 and levers 3742 or may be fixed if all pitchbends are to be done electronically. With a mechanical changer, thepedals 3741 and levers 3742 may be fitted with sensors that measure andconvert their displacement into control signals. Other control signalsmay be issued by a wrist operated expression wheel 3714 a (which inpractice works well for overall volume control), one or more pedals 3714b, some of which may be fitted with side-mounted spring-lever controls3714 c. The pedal rockers can be used for various control functions, butoffer a way of bending pitches electronically and holding them withouthaving to devote a foot or knee to that purpose. In contrast, theside-mount spring levers, as well as a spring loaded version of a rockerpedal, directly emulate the spring-return operation of a traditionalpedal steel guitar's pedals and levers. The pedals can also be used forthe introduction or variation of signal processing on one or moreselected strings and can be configured to make such effects whilesimultaneously changing pitch of selected string or strings. Theadaptation also provides for a strum-pad 3716 and miniature keyboard3721 in the picking area for use by idle fingers.

[0426] Finally, a foot-controller foot-switch unit 3742 for use inselecting stored programs, controlling signal processing, issuing notesor chords, operating drum machines, etc. is also provided.

[0427] 4.6 Sitars

[0428] The Sitar is an extraordinary rich instrument that is well-suitedfor the particular structural details of classical indian music. Itincludes a number of drone strings, only one or two of which can befretted in any musical way, a single melody string, and an octave pairof unfretted high pitch strings, called the “chikori” (Western spellingsvary) used for a variety of purposes including quite effective rhythmicaccents, all sharing a common sloping bridge that cause theaforementioned strings to twang to a degree determined by the slope ofthe bridge. A set of sympathetic strings with their own sloping bridge,which in some techniques can be arpeggiated and/or used as a small harpto a limited extent, is also provided.

[0429] The Sitar features a selected combination of both brass and steelstring types which have important essential distinctions in timbres Usesof the Sitar in Western music tend to fall into two categories: onewhere only the melody string, along with any sympathetic string action,is used, and another where the sitar's many drone strings force thetonality into the standard Indian tonal development system (rich andextraordinarily beautiful as it is). A Sitar-like sloping bridge hasbeen successfully put on a guitar (the Jerry Jones Coral Sitar” heard inmany Motown-era popular recorded songs), but all that remains is thetwang as the genius of the Sitar holistically has been omitted.

[0430] The invention provides for a powerfully rich adaptation of theSitar by combining the techniques described thus far with the signalrouting, processing, and synthesis techniques to be described later and,as with the previous examples, inherent aspects of the instrument.

[0431]FIG. 4 shows an adapted sitar as provided for in the invention.The core instrument is a standard Indian Sitar 400 with a standardmelody string 401.2, any one of a number of possible stringings of dronestrings -here two 401.3, 401.4, are used, and the chikori pair 401.5 a,401.5 b all sharing the common sloped bridge 402 a. Also part of thecore instrument, but not showed explicitly in the Figure for the sake ofclarity, are the sympathetic strings, typically eleven in number, withtheir own sloping bridge 402 b and multi-length termination area 402 onthe neck under the Sitar's curved frets. Because of the curved frets andthe flat bridge 402 a, the drone strings 401.3, 401.4 will not soundaccurately in pitch for nearly all the frets; further, thecharacteristic extensive bending of the string, often the interval of amajor fourth or more, requires quite a bit of area on the sitar neck(namely the entire bottom half of the neck) and as a result the dronestrings' use in melody is essentially nil. However, it is possible toadd an additional melody string 401.1 with its own lower bridge; thisstring would be pushed up to get the bending effects that the originalmelody string 401.2 is pulled down for. Both melody strings are providedwith fine tuning adjustments such as the typical beads 430; these mayalso be used on drone strings and/or alternate tuning mechanics may besubstituted throughout the instrument. Also provided, though notessential, is an additional set of strings 424 that can be used as aharp in addition to the limited-access aforementioned sympatheticstrings. In the Figure these are shown well-removed from the neck at thetop of the instrument, but the assembly 424 could be brought forward andput in the position currently occupied by the keyboard 421. The keyboard421 is also not essential but is easy to support electronically,employing the method associated with FIG. 8, in conjunction with thestrum-pad 416 a located on the plucking area of the neck and thestrum-pad 416 b located on the thumb-rest area of the neck; further,both the keyboard Harmonium and keyboard synthesizer are popular inIndian classical, modal, and Ghozal folk music. The strum-pads areprovided because of there particular potential use in the Rag (Raag,Raga) form both in laying out the scale of the Rag and melodyexpositional fragments which are repeatedly drawn upon.

[0432] Important to the adaptation is the pickup assembly 403 whichprovides a separate pickup for each melody string, each drone string,and either the chikori pair or its individual strings. The separateoutputs allow for pitch shifting of individual strings; in particular,the pitch shifted retunings of the drone strings and chikori can be madewhile playing. If the pickup is electromagnetic, the brass stringscannot be used. There is the opportunity here for alternative stringingsystems, particularly if pitch shifting of individual strings is used tocreate larger pitch-shifts, but the character of the brass strings isbeautiful and can be captured. One method is to use an optical pickupfor the pickup assembly 403. Another more radical approach is to replacethe sloping bridge 402 with a standard bridge arrangement fitted withindividual piezo pickups and to create the twanging using the off-bridgesitar plate discussed in association with FIG. 21.

[0433] The additional melody string can be tuned in union or in aninterval to the original melody string; because the have separate audiochannels they can be processed differently or be located at differentpositions in the stereo sound field. Further, the additional melodystring, strum-pads, and addition string assembly serve to expand animportant orchestrational aspect of seasoned Sitar technique, namely aconstant variety of timbres and effects with attention constantlyshifting among them. Finally, the electronic pitch shift retuningcapabilities allow for hitherto impossible tonality shifts within theSitar environment, while the electronic pitch shift pitch-bendcapabilities allow the drone strings to obtain pitch bending and themelody strings to be harmonized in a pitch-modulated manner.

[0434] It is also possible to carry simplified versions of the Sitartonality into more Western instrument formats. FIG. 38 shows an exampleflat-necked instrument 3800 with a five string section emulating a sitarstring arrangement and several additional strings used for bass or otheraccompaniment. The sitar emulation section involves strings3801.1-3801.5 sharing a common bridge 3802 a providing a separate piezopickup for each of the strings. The first four of these stringsterminate their vibrations at the nut 3802 c and are tuned with necktuners 3851, while string 3801.5 acts as a one-string fretted chikoriand terminates its vibration at one of the mid-neck frets. The bestterminating fret may depend on the chosen tuning and thus terminatingholes for this string might be provided at several different frets, forexample from the 4th fret to the 12th. String 3801.5 requires adifferent tuning arrangement and in order to accommodate a selection oftermination points the tuner 3802 may be located on the body, perhapswith the tuning head in a protective cut-out area of the body. Anotherimportant item is that string 3801.4 is brass while the others aresteel, creating the Sitar sonority. The additional strings, shown inFIG. 38 as six in number 3801.6-3801.1, can serve as bass strings orharp strings and may be terminated on a rapid return bridge (such as theHipshot Trilogy product) to facilitate flexible use during performance,particularly because of the width of the fretted neck; alternativelythese strings may be unfretted as on the European arch-lute. Thesestrings are most likely best served by steel strings and thus eachstring may be given its own pickup either at the rapidly retunablebridge assembly 3802 b or with a body-mounted multi-channel pickup 3805.

[0435] Finally, in lieu of a sloped bridge or the arrangement of FIG. 8,it is also possible to create a synthetic and/or enhance the actualsympathetic/buzz/twang aspect of these instruments with signalprocessing techniques. A precisely-set, short audio delay (for example 3msec, 6 msec, or 12 msec for various octaves of the note E in A440tuning) with high positive feedback acts as a resonator that easilydistorts when excited by an audio input at its resonant frequency. Thisresolutes in a swelling and diminishing twanging similar to that createdby the sloping bridge. This sort of effect can be found in one of theexample presets of the Boss SE-70 stereo signal processor. The effectcan be considerably enhanced by following this resonator with alow-speed sweeping flanger and a low-speed sweeping chorus, each withtheir own moderate amount of positive feedback, and made moreemotionally powerful by low-speed auto-panning location modulation. FIG.39 shows an example multiple-pitch sympathetic/buzz/twang resonatorfeeding an input signal 3900 into banks of short audio delays3901.1-3901.n with high resonances tuned to each selected pitch, eachfollowed by a dedicated low-speed sweeping flanger with moderateresonance 3902.1-3902.n, a dedicated low-speed sweeping flanger withmoderate resonance 3903.1-3903.n, and a low-speed auto-panner3904.1-3904.n. All autopanner outputs are summed by a stereo mixer 3905a, 3905 b to create a stereo output 3906 a, 3906 b. It is important thatall the sweep oscillators be slightly detuned to minimize repetitivediscernible patterns. The number of individual resonant pitchessupported would involve every note in the scale and perhaps an octave ofsome selected notes such as the tonic and fifth; this could range from 5to 16 pitches. It is understood that variations, simplifications, andelaborations of this example arrangement are possible.

[0436] 4.7 Pipas

[0437] Like the Indian Sitar, the Chinese Pipa features a mix of stringtypes, here involving steel, silk, and composites of these. The Pipa(and to some extent its Japanese colleague, the Biwa) also has a richancient tradition yet contemporary appeal. Despite being far less known,it is capable of a great range of sonic techniques, with a high numberof formal playing techniques as compared to many other instruments.Included in the extensive technique suite are a number of body taps andimpacts made on the large front surface of the instrument.

[0438] As with the above example adaptations, the invention provides foradaptations of the Pipa that involve instrument elements of theinvention set to capture and complement the characteristics of this richand deep instrument. Again, piezo bridge pickups are felt to be the bestmode for capturing the subtle acoustic nuances of the different stringtypes, and a separate pickup for each string permits the usualmulti-channel signal processing possibilities and control signalextraction for controlling synthesizers, signal processing, lighting,and special effects. Body taps and impacts can be directed towardsimpact sensors, and the usual possible collection of extra strings,keyboards, strum-pads, touch-pads, sliders, switches, buttons, sensors,etc. may be added to the large open area for instrument augmentations.In particular, strum-pads and a bank of harp strings are especiallyapplicable due to the common use of pentatonic scale sweeps and repeatedshort melodic figures during development. Also especially useful forincorporation into Western sonic structures would be the addition of abank of bass strings and the use of signal processing as the Pipa tonalrange, though fascinating, arrives somewhat unfocused on undevelopedWestern ears unfamiliar with the instrument repertory.

[0439]FIG. 40 shows an example adapted Chinese Pipa as provided for bythe invention featuring various impact sensor arrays 4003 a, 4003 b,4003 c, a keyboard 4004, slider array 4007, switch array 4008, touch-pad4009, strum-pad 4010, and separate piezo pickups for each string4001.1-4001.4 at the bridge 4002.

[0440]FIG. 41 shows another example adapted Chinese Pipa as provided forby the invention featuring impact sensors arrays 4003 a, 4003 c,strum-pad 4010, an unfretted harp string array, involving a terminatingbody tuning bridge 4110 a and body bridge 4110 b, and an unfretted bassstring array involving body bridge 4111 a and head tuning bridge 4111 bwith tuners 4113. The two string arrays may have individual string orgroup pickups located either in the body bridges 4111 a, 4111 b and/oron the body at an interior portion of the string vibration 4114, 4115.

[0441] It is understood that many other combinations of instrumentelements are possible.

[0442] 4.8 Erhus, Dilruba, Esraj, Sarangi, Kamamcheh

[0443] Each of these bowed instruments has its own rich tradition andspecial tonal qualities. Many of these instruments are used to accompanyvocals or even to replace a singer due to the vocal quality of theinstrument.

[0444] The invention provides for adaptation of these instrumentsinvolving instrument elements of the invention set to capture andcomplement the characteristics of the traditional instrument and itsmusical traditions. In particular, in addition to the vocal quality ofthe sounds, bowing is a more conspicuous part of the sound as opposed toWestern bowed instruments which encourage burying the perception ofbowing logistics in favor of overall smoother tones.

[0445] Again separate pickups may be used for each string:electromagnetic, piezo, and/or optical as appropriate for the type ofstring material, mounting arrangements, and other engineeringconsiderations. A separate pickup for each string permits the usualmulti-channel signal processing possibilities and control signalextraction for controlling synthesizers, signal processing, lighting,and special effects. Those instruments with sympathetic strings, such asthe Esraj, Dilruba, and Sarangi, may also include pickups for thosestrings as described in previous example instrument adaptations.

[0446] Because each string has its audio channel picked up intimatelywith the string, it is possible to diminish some effects of the bodyresonance and replace it with electronically created resonances. Inparticular, vocal sounds are known to appeal to the ear as vocal innature due to the relative center frequencies of a pair of predominantresonances as illustrated in FIG. 17. Through electronic synthesis ofthese resonances the vocal character of the instrument can be changedand, in fact, varied over time if one would dare to make the vocal bowedinstrument literally sing.

[0447] Further, because of the somewhat different role of bowing, moreattention can be paid to collecting control information from the bow.However, the bow sensor techniques described can also be used to greatadvantage in Western bowed instruments. FIG. 42 shows a bow fitted withsensors to gather information from the hand, bow hairs, and bow motion.The bow 4200 fitted with bow hair 4201 as usual, may include a bow-hairtension sensor 4202, a free finger pressure-sensor and/or null/contacttouch-pad (or pressure-sensor array) 4203, a handle grip 4204pressure-sensor 4205, and an internal accelerometer 4206 to determinebow direction and acceleration. These control parameters can betransmitted to the instrument by attached electrical or fibre cable,wireless optical, wireless radio, or other means. As a simple example,the free finger controls the choice of resonant vowel formants for theinstrument while the hand grip pressure 4205 or pressure on the freefinger control 4203 may be used to control the sympathetic stringamplitude or a signal processing parameter. The bow tension andaccelerometer measurements can be used to control emphasis signalprocessing or darkening lowpass filtering.

[0448] 4.9 Flutes and Recorders

[0449] Reed instrument layouts have been used in wind controllerproducts by Akia and Yahama. However, flute-like (embouchure air hole)and recorder-like (fingers normally down) instruments have to date notbe used as models or methods for electronic instrument controllers. Itis noted that some types of Western flutes have at least some openholes, many folk and non-Western flutes have only open holes, and someflutes and recorders have at least one hole that is open but is operatedby a levered key. In the discussion below, the flute example isconsidered to be purely closed hole and key operated while the recorderexample is considered to be purely open hole without levered keys; thethus illustrated techniques can be freely applied to other hole andlever arrangements of a particular instrument variant. FIG. 43 showsadaptations of a Western (closed-hole) flute and an (open hole) recorderlayouts with pressure-sensors, or small pressure-sensor arraytouch-pads, replacing sites of the keys, together with air turbulencemeasurements, and air pressure average measurements as provided for inthe invention. In either instrument adaptation, the instrument may bemaintained as a sounding instrument or used as a model for an allelectric controller. In a sounding version, an attached or internaltransducer may be used to provide both an audio signal for processingand a means for note event control extraction (for example, usingpitch-to-MIDI technologies such as those of, or superseding, the RolandCP-40).

[0450] In the example flute and example recorder shown in FIG. 43, thebody area around the flute embouchure wind opening 4301 and recorderwind opening 4351 may experience air pressure changes and turbulencewhich may be measured with sensors and signal processing as describedearlier; if the sensors and wiring to them are mounted securely in lowprofile the instrument behavior and sound will not be noticeablyaffected. Alternatively, in non-sounding versions, more obtrusiveinternal sensing of air turbulence and/or air pressure may be employed.

[0451] In a sounding adaptation of the closed hole flute 4300, the areaof the keys which contact the fingers 4302 can be covered with simpleswitches, a pressure-sensor, or a pressure-sensor array. Alternatively,in a non-sounding controller adaptation of the closed hole flute 4300,the area of the keys which contact the fingers 4302 can be replaced bysimple switches, a pressure-sensor, or a pressure-sensor array. In thecase of the open hole recorder, simple switches, a pressure-sensor, or apressure-sensor array can be put around the perimeter of any of eachsingle-hole 4353, each double-hole 4354, and the thumb-hole 4355.Because of special playing techniques associated with the double-holes(i.e., “half-covering”) and thumb-hole (thumb tip flip or other“half-covering” methods), these areas may be handled with morespecialized switch and/or sensor arrangements.

[0452] For the most part such hole-positioned and key-positioned sensorsmay be used to assist in issuing note events but ranges of additionaltechnique can be developed for more sophisticated control. A lesstechnique-oriented approach would be to put simple switches, apressure-sensor, or a pressure-sensor array in an area 4306, 4356 wherea thumb is otherwise only used for supporting the instrument.

[0453] As with the other instrument examples, it may also beadvantageous to place additional instrument elements such as strum-pads,touch-pads, sliders, switches, buttons, other sensors, etc. on the bodyof the instrument.

[0454] 4.10 Gongs, Bells, Cymbals, Chime Bars, other Metallaphones, andAcoustic Drum Heads

[0455] Gongs, bells, cymbals, chime bars, xylophones, and othermetallophones, as well as the stretched heads of acoustic drums, can beproblematic to amplify because they typically undergo significantdisplacement motion when struck yet their sound may alter significantlyif this motion is restrained and/or if a surface transducer is attachedto them. It is noted that there are many types of musically usefulnon-stereotypical gongs with widely varying timbres, including forexample the non-crashing, pitched Indonesian gongs with close-setovertones which beat at low frequencies creating a complex tremoloeffect that sounds in many of these instruments very similar to pitchvibrato.

[0456] The invention provides for quality audio signal capture fromthese types of instruments because of their musical usefulness, the richpossibilities for signal processing their sounds, and the visual appealof their playing in a performance situation. FIG. 44 shows how anoptical pickup may be created for a suspended gong; this technique mayalso be used for many other types of metallophones and acoustic drumheads. In the example, a gong 4400 is supported by small ropes 4401 athrough holes 4401 b or other means. Etched into, polished into, sprayedonto, adhered onto, etc. the gong 4400 is a reflective or refractivearea 4404 which may be unstructured or may include a reflective patternor refractive properties.

[0457] One or more, typically two, light sources 4402 a, 4402 b,typically coherent and/or modulated, direct light beams 4403 a, 4403 bto the reflective or refractive area 4404. The materials, light sources,and geometry are arranged so that the return light path 4405 ismodulated in amplitude and/or intensity when the gong vibrates at audiofrequencies. Reflective or refractive light 4405 from the gong isdirected at one or more light detectors 4406 which use the resultinglight amplitude or intensity modulation to create an audio signal.Lowpass filters are used to remove subsonic signals resulting from theswaying of the gong. FIG. 45 shows an example metal bar which can bemounted so that extremes of impact displacement are relatively confined.In this arrangement the bar 4450 is suspended by small ropes 4451 athrough holes 4451 b or other means. The vibration of the bar may becaptured in a number of ways; illustrated are a similar optical sensor4452, and electromagnetic pickup 4453 should the bar 4450 beferromagnetic or have a ferromagnetic plating or attachment near thepickup, and a capacitive plate 4454 which can be used to measure thevarying capacitance between the plate 4454 and the bar 4450 via wires4455 a, 4455 b connected to the plate 4454 and a conducting support rope4451 a or other means. Similar methods may be employed for acoustic drumheads. Not shown is the case where a bar or tyne is tightly secured onone end; in this case not only may the bar be amplified by similarmethods but also by a piezo sensor in the support as in the case of theMbira discussed earlier.

[0458] It is noted that these pickup strategies all pick up localizedvibrations from the metallophone. As with instrument strings, theproduced timbre will vary widely with the selected pickup area. It istherefore provided for in the invention that multiple pickup areas maybe used, permitting multi-channel signal processing to be applied to asingle gong in a way like that described earlier for instrument strings.

[0459]FIG. 45 shows example gong arrays as part of a one-hand ortwo-hand percussion instrument stand 4500. The top tier of gongs4501.1-4501.n in this example are shapely gongs such as the Indonesiangongs described earlier. The middle tier of gongs 4502.1-4502.m in thisexample are flat gong plates such as those found in Chinese percussion.Also shown in this example instrument are two sets of acoustic drums4603 configured for one-hand rolls as described earlier for electronicpads; the vibration of the heads of these drums are captured as per thedescribed methods for gongs and bars.

[0460] 5 Alternative Audio and Control Signal Sources

[0461] Historically new instruments have been created throughincorporation of not only newly developed technologies but also newlydiscovered phenomena. In this section recently available understandingof largely unrecognized or unutilized processes are adapted by theinvention for use in generation audio and/or control signals.

[0462] 5.1 Chemical Oscillations, Patterns, Waves, and Rhythms

[0463] The Belousov-Zhabotinskii reaction [Tyson] and many similar“non-equilibrium” chemical reactions exhibit oscillatory and animatedpattern-forming wave propagation and mathematical chaos effects whichcan be visually and electrically monitored [Gray, Scott]. Thesebehaviors are the result of nonlinear dynamics governing the evolvingreactant concentrations varying within the mixture over time [Nicolis].Varying types of electrodes can be used to measure component reactantsindependently. If multiple electrodes are used, differing but correlatedwaveforms are produced simultaneously, useful for both control andspatial timbre formation methods described later on. To some extentthese reaction processes may also be controlled [Ruoff; Nagy-Ungvaraiet. al.] via electric fields, reactant modulation, etc.—means that infact can be controlled directly or indirectly by electrical signals.

[0464] Chemical indicators may be used to visually enhance theobservable contrast of pattern animation [Tyson; Orban et. al.]. Theresulting animated patterns, which range from swirling spirals tocomplex tidal forms—remnants of 1960's animated hallucinogeniciconography—can be captured by video camera. The character of thepatterns have visual and intuitive appeal and familiarity because theyreadily occur in biology, geology, and other parts of nature [Nicolis,Baras]. Populations of these chemical systems can be coupled by variousmeans and as thus are observed to have rhythmic and turbulent behaviors[Kuramoto]. These various dynamical properties of non-equilibriumchemical reactions can be adapted to create a new exciting class ofinstrument entities and performing environments which are describedherein.

[0465] 5.1.1 Chemical Oscillators as Sound Sources

[0466] In their simplest form, these chemical reactions act largely assimple oscillators [Tyson;

[0467] Gray, Scott]. The oscillations are the result of nonlineardynamics governing the evolving reactant concentrations varying withinthe mixture over time and typically are in the form of slowly evolvinglimit cycles [Field, Noyes; Gray, Scott]. Each reactant-monitoringelectrode then produces an oscillatory signal for the duration of theoscillatory concentration variation of that reactant.

[0468] In practice most oscillations occur at very slow rates, forexample with a period of 40-60 seconds, and have a short life time, forexample under a hundred cycles, unless reactants are refreshed. Thedesign of wider ranges of chemical oscillators has been investigated[Epstein; Epstein] and in that it is conjectured that chemicaloscillations may drive insect wing vibrations it may be possible todesign triggered chemical oscillators that oscillate at audio rates withvarious oscillatory durations. Such chemical reactions, whenelectrically monitored, can be used directly as sound sources in thesame manner as an electromagnetically-monitored orpiezo-mechanically-monitored guitar string.

[0469] Less speculatively, recorded measurements of known slowshort-lived oscillatory chemical reactions [Gray, Scott] may be capturedand processed as “audio samples” which can be pitch-shifted and splicedfor arbitrary duration with conventional audio sampling technology.Further, mathematical models of these oscillatory behaviors [Field,Noyes] can be numerically simulated and altered so as to change rate,duration, and other attributes [Wang, Nicolis] as per model-based audiosynthesis. Such numerical models then add a new non-acoustic class ofmodeled elements to the well establish acoustically vibrating ones suchas strings, pipes, tynes, membranes, etc., and as with theacoustic-based models, can be adapted and extended to create yet othernew effects.

[0470] 5.1.2 Chemical Patterns as a Dynamic Controller

[0471] The inherent time scales of visual and electrically measurablepattern evolution in most of the well-know non-equilibrium chemicalreactions, along with their potential for direct and indirect electricalcontrollability, makes these non-equilibrium chemical reactionsinteresting candidates for the generation of control signals. Theinvention provides for the spatial patterns of these non-equilibriumchemical reactions to be measured and converted into control signals andpotentially, with any of several chemical processes, to control viacontrol signals and/or to video capture for display or recording. Theinvention provides that these measured control signals may be used tocontrol any one or more of note events, signal processing, lighting, orspecial effects.

[0472] The invention provides for spatial patterns of thesenon-equilibrium chemical reactions to be measured electronically byspecific types of electrodes [Gray, Scott] and/or via a video cameracombined with image analysis, parameter extraction, and control signalassignment. If electrodes are used, these may be of various types,including those responsive to variations in specific families of ionconcentrations [Gray, Scott] as well as those used to measure electricfields, potential differences, electrical resistance, etc. Theseelectrodes may be distributed spatially in one, two, or the dimensions.FIG. 46 illustrates spatial arrays of electrodes which may be used formeasurement, as well as control, in two-dimensional andthree-dimensional configurations. In an example two-dimensionalconfiguration, a shallow vessel 4610 has its bottom surface fitted witha two-dimensional array of electrodes 4611 selected from among the typescited earlier. These electrodes connect by wires 4612 to interfaceelectronics 4613 that extracts the measurement information, from thisextracts parameters (potentially under stored program control), and thenassigns these parameters (potentially under stored program control) tooutgoing control signals 4614. Some of the electrodes 4611 may bemeasurement only, some may be control only, and some may be dual-use.Any control capabilities and/or stored program recall may be controlledby incoming control signals 4615. In an example three-dimensionalconfiguration, a volume vessel 4620 has its bottom surface fitted with athree-dimensional array of electrodes 4621 selected from among the typescited earlier. These electrodes connect by wires 4622 to interfaceelectronics 4623 that extracts the measurement information, from thisextracts parameters (potentially under stored program control), and thenassigns these parameters (potentially under stored program control) tooutgoing control signals 4624.

[0473] Some of the electrodes 4621 may be measurement only, some may becontrol only, and some may be dual-use. Any control capabilities and/orstored program recall may be controlled by incoming control signals4625. Also illustrated in the three-dimensional example are an examplereactant intake tube 4632 and example reactant outtake tube 4634 throughwhich new reactants 4631 may be introduced and old reactants 4633 may beextracted; this arrangement can be used to vary the relativeconcentrations of chemicals in the vessel or maintain balance throughreactant refreshing. The exchange rates and relative concentrationslevels can be electrically controlled by conventional chemistryinstrumentation means and thus the invention provides for these to bepotentially controlled by incoming control signals. It is understoodthat multiple intakes and outtakes such as 4632, 4634 may be utilized,and that the intakes and outtakes may have interior spatial distributionstructures within the vessel 4620. It is also understood that thesemethods would also be applicable for one-dimensional or two-dimensionalarrangements and other vessel shapes such as dishes, tubes, etc.

[0474] It is also possible to measure the evolving chemical patternswith a video camera, particularly when differentiating visual indicatorcompounds [Tyson; Orban et. al.] are introduced into the mixture. FIG.47 shows an arrangement where evolving chemical patterns 4700 in thedish 4610 of FIG. 46 are illuminated with light sources 4701 andvisually monitored by an overhead camera 4702; here the light sourcesand camera are supported by arbitrary stand methods 4703. The inventionprovides in particular for the video camera 4702 to be combined withsimple to complex real-time image processing operations such as positionthreshold detection, edge detection location, pattern detection, patternarea measurement, pattern rate of change, etc. to derive a multitude ofparameters which may be mapped into control signals. Because visualchemical indicators can be used to identify localized concentrationslevels of specific chemicals, the aforementioned pattern recognitiontechniques may in some configurations conceptually be used in place ofthe potentially more costly chemical-measurement electrode arrays andsupporting electronics.

[0475] The invention provides for the aforementioned arrangements to beused as an interactive chemical performance environment. Outgoingcontrol signals generated by the spatial chemical patterns may be usedto control any one or more of: note events, timbre modulation, lighting,and special effects. Incoming control signals provided by or extractedfrom audio signals, electronic instrument elements, real-timesequencers, actuators, video cameras, or body position indicators(gestures, dance, stage position) can be used to control the evolutionand influence the shapes of the chemical patterns. Video of the patternsmay be displayed on monitors or projected, via video projector, onto thestage area behind, above, or on one or more performers. The projectedvideo image may be actual or processed by video signal processing (forexample, changing color maps, contrast, solarization quantizationthresholds, etc.) which in turn may be controlled by control signalsgenerated in realtime by the performers. In this manner, one or moreperformers may interactively perform with music, sound, and visualeffect with a non-equilibrium chemical reaction environment.

[0476] It is also possible to numerically or electronically simulate thechemical dynamics on a computer, generating similar types of controlsignals, visual output, and interactive control capabilities. Thismethodology is discussed in more detail. Because numerical andelectronic simulation can generalize the process beyond physicallimitations, in principal a broader range of interactive dynamics wouldbe made possible by this method. However, the excitement and charm ofinteracting with a live chemical process is difficult to entirelyreplace with a computer program.

[0477] 5.2 Photoacoustic sources

[0478] Photoacoustic phenomena is a relatively new area of study.Although most of the gathered knowledge and work in progress is largelyoriented to probably inapplicable areas relating to, for example,non-destructive testing, there are a few phenomena, such as lightstimulated acoustic emissions and the modulation of light throughvibrating transparent or translucent materials that can be developed formusical purposes [Lusher; Murphy et. al.; Bicanic, Dane]. The inventionprovides for the incorporation of these, particularly in that light canbe used as part of performance and visually recorded material.

[0479]FIG. 48 illustrates example optical measurements of photoacousticphenomena in applicable materials 4900, 4910 which may be converted toelectrical signals, and an example electro-acoustic measurement ofphoto-induced acoustic phenomena in applicable materials 4920. Examplephotoacoustic materials 4900, 4910, 4920 may be in gas, crystal, liquidcrystal, plastic, elastic, fluidic or other forms.

[0480] For material 4900 which emits light in response to acousticvibration, a light sensor 4901 may be used to recover the light emissionevent. For material 4910 which modulates light in response to acousticvibration, a light sensor 4901 may be used to recover light provided bya light source 4902 which is directed through the material 4910. Formaterial 4920 which emits acoustic vibration in response to light, anelectro-acoustic sensor 4921 may be used to sense acoustic vibrationemitted in response to one or more appropriately positioned lightsources 4902 a, 4902 b.

[0481] In the above, it is noted that ultra-sonic vibration, even up toa few hundred Khz, is still potentially useful as these signals may bepitch-shifted or heterodyned down to audio ranges.

[0482] 5.3 Electronic/Numerical Dynamical System and Relational SystemSimulation

[0483] Electronic and/or numerical algorithm methods may be used toimplement mathematical dynamical models including mechanical vibration,fluid mechanics, stellar evolution, biological processes, etc. as wellas abstract non-equilibrium, fractal, and chaos process models. Suchmethods are already in place in the synthesis of musical sound vibrationprocesses modeling conventional musical instruments, for example, inmodel-based sound synthesis as used in the Yahama VL1.

[0484] Because numerical and electronic simulation can generalize theprocess beyond physical limitations, in principal a broader range ofinteractive dynamics and real-time measurements of them would bepossible as compared to that which could be obtained in the real-worldunder realistic conditions. Further, electronically or numericallymodeled processes may be time-scaled so as to produce audio frequenciesor more slowly evolving control signals. The invention provides for theuse of such electronic and/or numerical algorithm method so as toimplement mathematical dynamical models of adapted real-world orabstract processes. Incoming control signals can be used to selectand/or affect the structure and/or parameters of the modeled dynamicsand/or relations, and the modeled dynamics and/or relations may be usedto create any one or more slowly varying outgoing control signals,visual image signals, or direct audio signals.

[0485] Examples of abstract processes may include interactive navigationthrough a fractal structure, the fractional integration of an audiofrequency square wave as it evolves into triangle and parabolicwaveforms, etc. Examples of real-world models rich in semiotic value forperformance may include adaptations of interactive control of galacticinteraction dynamics, language models, etc. as well as the use ofliterary plots, classical mythologies, etc. which have been used bycomposers for centuries (i.e., Monteverdi's Orfeo, Strauss' Electra,Stravinsky's Odepus Rex, etc.)

[0486] 5.4 Environmental

[0487] Earlier instrument elements and instrument entities associatedwith environmental aspects of stages, rooms, and the outdoors weredescribed. Examples of this include the tracking of the position and/ormotion of performers, the tracking of artificial fog cloud migration,room internal and outdoor meteorology, and audience motion activity. Asindicated in those discussions, these may be used, to the extentartistically applicable, to generate control signals for the control ofnote events, signal processing, lighting, and special effects.

[0488] 6 Generalized Instrument Interfaces

[0489] Referring to FIG. 1, it is recalled that the invention providesfor both instrument entities 100 and signal routing, processing, andsynthesis entities 120 to be fitted with compatible electricalinterfaces, termed generalized instrument interfaces or (or moreconcisely, generalized interfaces) 110, which can exchange any of thefollowing:

[0490] incoming electrical power (111)

[0491] outgoing control signals from switches, controls, keyboards,sensors, etc., typically in the form of MIDI messages but which may alsoinclude contact closure or other formats (112)

[0492] control signals to lights, pyrotechnics, or other special effectelements within and/or attached to the instruments, said signals beingeither in the form of MIDI messages, contact closure, or other formats(113)

[0493] outgoing audio signals from individual audio-frequency elementsor groups of audio-frequency elements within the instruments (114)

[0494] incoming excitation signals directed to individualaudio-frequency elements or groups of audio-frequency elements withinthe instruments (115)

[0495] outgoing video signals (such as NTSC, PAL, SECAM) or imagesignals sent from the instrument (116)

[0496] incoming video signals (such as NTSC, PAL, SECAM) or imagesignals sent to the instrument for purposes such as display or as partof a visually controlled instrument (117).

[0497] The interfaces may be realized by any one or more of: connectors,cables, fibers, radio links, wireless optical links, etc., individually,in combinations, or in or sequences of these.

[0498] In most envisioned realizations this interface would be involveone or more connectors fitted with driving and/or receiving electronics,and the connectors on instrument entities 100 and signal routing,processing, and synthesis entities 120 would be connected by a pluralityof wires in either balanced or unbalanced transmission mode.Alternatively one or more coax cables, fiber optic cables, radio links,wireless optical links, etc. may be used to replace part or all of theplurality of wires. Any of these approaches may use any of a variety ofmultiplexing techniques [frequency-modulated and/or phase-modulatedand/or amplitude-modulated carrier, wavelength-division, time-division,carrier-less constellation synthesis (such as CAP), statistical, etc.)individually or in combination to reduce the number of partitionedphysical signal channels (wires, fibers, radio channels, wavelengths,etc.].

[0499] When these generalized interfaces are realized via one or morephysical cables (electrical, optical, etc.), some realizations may use asingle connector for fully spanning generalized applications while otherrealizations may consist of an ensemble of connectors in a functionalsplit so as to handle particular organization, expansion, and/orevolutionary needs.

[0500]FIG. 49 shows how generalized interfaces can be built in whole orvia separable parts which may be used selectively as needed orappropriate. In a single connector all-purpose approach, a single cable4901 carries all the signals allowed for in the generalized interface upto a predefined number of maximum instances. Referring to FIG. 1, allinstrument entities 100 and signal routing, processing, and synthesisentities 120 would provide mating connectors to that of 4902. In oneimplementation, the cable 4901 and connector 4902 provide a relativelylarge number of separate interconnecting wires, for example twenty tothirty. Alternatively, in a future preferred arrangement employing thenlow-cost standard signal multiplexing and/or directional multiplexingtechniques, this cable 4901 could be a simple one or two fiber optic orcoax cables potentially supplemented with two to three power conductors;alternatively, if any coax cables are used to carry signals they canalso be used to simultaneously carry power on the same conductors.Further, if each instrument is able to provide its own power by means ofwall plugs and/or batteries, the aforementioned implementations need notinclude any power carrying capabilities and associated conductors. Inthis arrangement it will be further possible to use radio and/or opticalwireless channels to carry the signals among instrument entities 100 andsignal routing, processing, and synthesis entities 120. In thisarrangement then, a common multiplexed incoming and outgoing signal,potentially itself directionally multiplexed on the same channel, can becarried interchangeably by optical fiber, coax cable, wireless radio, orwireless optical transmission mediums.

[0501] Alternatively, it is possible to functionally partition thegeneralized interface into standardized component interfaces which maybe served by separate connectors. A multi-connector “Hydra” cable can beused to provide selected groups of two or more of these standardizedconnectors, including a “fully-populated” Hydra cable with all thedefined connectors. If only one connector of the several defined ones isneeded, then a single connector cable may be used if desired; for thisreason, it may be desirable to assign connectors to the functionalpartitions which are standardly available on mass-produced singleconnector cables. FIG. 49 shows an example of this arrangement. Afully-populated Hydra cable 4911 may include the following partition andassigned connectors:

[0502] Two “single-channel” group audio outputs (unbalanced, dedicatedground) 4912; for example, a TRS male plug.

[0503] Ten or twelve multi-channel audio outs (unbalanced, dedicatedground) and audio power 4913; for example a 13-pin DIN male or HDB15-pin VGA male (with 14 pins actually populated and connector shellsinterconnected).

[0504] One video out channel and one video in channel (unbalanced,dedicated ground) and incoming video power 4914; 6-pin DIN male.

[0505] One balanced MIDI out channel and incoming MIDI power (onnon-MIDI pins) 4915; 5-pin DIN male.

[0506] balanced MIDI in channel and incoming controlled element power(on non-MIDI pins) 4916; 5-pin DIN male.

[0507] Six excitation drive channels (unbalanced, dedicated ground)4917; 8-pin DIN connector (only six pins need if the signal routing,processing, and synthesis entity 120 creates the relatively high-powerdrive signals, while eight pins allow provision of power to theinstrument entity 100 for internal high-power signal generation).

[0508] Further, any connectors not served by a given Hydra cable and/orexpansions to support additional channel-carrying needs may be supportedwith additional cables:

[0509] It is understood that the aforementioned as explained andillustrated in FIG. 49 are merely examples; other arrangements arepossible.

[0510] 7 Signal Routing, Processing, and Synthesis

[0511] The general principals for the architecture of the signalrouting, processing, and synthesis entity 120 as provided for by theinvention include all or a significant number of the following:

[0512] flexible multi-channel handling of audio, control, and videosignals

[0513] a hierarchically modular control and control signal routingstructure

[0514] course to very fine-grain control signal routing (for example, inthe context of MIDI, routing at the MIDI port level, routing the MIDIchannel level, and routing at the individual note number and continuouscontroller number levels)

[0515] the incorporation of mixing in audio routing and message mergingand polyadic operations in control signal routing

[0516] control signal extraction from audio and/or video

[0517] audio signal, control signal, and potentially video signalprocessing

[0518] audio signal, control signal, and potentially video signalsynthesis

[0519] real-time control signal/event replay all under extensivereal-time control.

[0520] 7.1 Audio signal routing

[0521] Audio signal routing is provided for in the invention by bothswitching and mixing functions. Switching functions may be realized asstored program cross-bar switches. Mixing functions may be provided inthe form of possible multiple-input multiple-output mixing matrices andan additional final mixing stage may include some dedicated signalprocessing functions. Mixing functions provided for in the invention areadvantageously controlled in real-time by control signals.

[0522] Functional examples of the functionality provided for in saidmultiple-input, multiple output mixing matrices is that of the SoundSculpture model Switchblade MIDI-controlled mixer (but empowered with asignificantly larger number than two MIDI continuous controller inputs)or the Peavey PM-8128 (but provide with additional inputs and outputs).Functional examples of the functionality provided for in said finalmixing is that of the Yahama DMP MIDI-controlled mixer models,particularly the DMP 9-16 (but with additional presets). In theinvention, the mixing and switching functions are preferably anintegrated component within a larger-scale hardware and softwareconstruct rather than an off-the-shelf module.

[0523] 7.1.1 General audio switching and mixing

[0524] Referring to FIGS. 1-2, input signals directed to audio routingand mixing may include the audio outputs from the instrument entities100, outputs from audio signal processing elements 125, and audio signalsynthesis elements 129 a. Still referring to FIGS. 1-2, output signalsdirected from audio routing and mixing may include audio inputs toexciter elements within instrument entities 100, audio signal processing125, control signal extraction 128 a, and the overall audio outputs toamplification and/or recording facilities.

[0525] 7.1.2 Multi-channel Audio Signal Handling

[0526] The invention provides for extensive support for and exploitationof multi-channel audio signals from instruments with multiple vibratingelements.

[0527] Multi-channel transducers have been used inmultiple-vibrating-element musical instruments; these uses appear to beconfined to guitar synthesizer interfaces (as with the Boss GP-10),individual adjustment of each vibrating element mix level (as with theGibson Chet Akins guitar), and creation of panned stereo mixes (Biaxpickup, Passaic synthesizer interface, Turner string pan-pot guitar, VanHalen-endorsed guitar with right/left switches for each string). Thesesimilar approaches may be generalized by a common diagram. FIG. 50 showsmultiple vibrational elements with multi-channel transducers applieddirectly to stereo or multi-channel mix-down. Each of a plurality ofvibrating elements 5001.1- 5001.n are is coupled by appropriate butseparable means to each of a plurality of transducers 5002.1-5002.n,each producing electrical signals which are applied to either singlechannel or stereo mixing circuitry. In the case of the Gibson ChetAtkins electric-acoustic guitar, the mix is monaural and as such isequivalent to delivery only through output A 5004 a, with noimplementation of output B 5004 b. In the case of the Biax pickup, the“pan” position in the stereo field is hard-set to either output A 5004 aor output B 5004 b. In the case of the Turner guitars, the “pan”position in the stereo field is set between output A 5004 a or output B5004 b via adjustable potentiometers.

[0528] Specifically the invention provides for bringing the signals frommulti-channel transducers 5002.1-5002.n to individual signal processingstages 5005.1-5005.n before mixing, allowing far more extensivecapabilities to be created. FIG. 51 shows multiple vibrational elementswith multi-channel transducers and individual signal processing prior tomixing. FIG. 51 highlights the functional signal processing distinctionfrom existing commercial products generalized by FIG. 50. Thisrelatively simple conceptual (though potentially hardware and/orsoftware intensive) change makes a number of extraordinary thingspossible:

[0529] Conventional pitch-shifting signal processing can be used on eachstring signal to create:

[0530] “generalized pedal steel guitars” (augmenting or replacingmechanical pedal tuning changers with pedal, lever, spring-wheel,optical, or other electronic controls determining pitch shift amount)

[0531] instantly retunable guitars (augmenting or replacing mechanicaltuning changers such as the Hip-shot “Trilogy”)

[0532] a true electronic simulation of so-called “multi-course”instruments (such as a 12-string guitar, mandolin, lute, etc.) whereindividual elements making up the “multi-course” are simulated usingpitch shifting to create either octaves or slightly mis-tuned unisons

[0533] multi-key, multi-modal Indian sitars; here drone and sympatheticstrings can be electronically retuned while playing, allowing a moreflexible and robust mix between Eastern (fixed tonality) and Western(modulating tonality) musical forms.

[0534] multi-key, multi-modal African mbiras, African koras, Japanesekotos etc.; here fixed pitch vibrating elements (tynes, strings) can beelectronically retuned while playing, allowing a more flexible androbust mix between Eastern (fixed tonality) and Western (modulatingtonality) musical forms.

[0535] spatial-spectral animated instruments where individual vibratingelement sounds may be location modulated within a stereophonic or otherspatial sound field (using low-frequency sweep chorusing, continuousauto-panning, etc.).

[0536] separate distortion circuits for each vibrating element, forexample, to create:

[0537] powerful guitar chord sounds previously obtained only by multipleinstruments or multi-track recording)

[0538] simulated sitar-bridge effects (using the methods to be describedin conjunction with FIG. 39)

[0539] finely frequency-equalized instruments where different frequencyequalizations are applied to each vibrating element.

[0540] mixed timbre instruments where different signal processingmethods are applied to each vibrating element.

[0541] (More recently, a functionally limited—although verytechnologically progressive—version of the signal processing approachillustrated in FIG. 52 has since appeared in the Roland VG-1 COSM guitaremulation product.)

[0542]FIG. 52 shows addition of a control signal extraction element tothe arrangement of FIG. 51. As shown in FIG. 52, the invention providesfor signal outputs from the multichannel transducer(s) to be fed inparallel to a control signal extractor 5010 so as to simultaneouslyissue control signals 5011 to any one or more of music synthesizers,signal processors, lighting, special effects, etc. The invention thereinprovides for an operating mode of control signal extraction where eachindividual vibrating element to have associated with it one or moreindividual control signals. The resulting system, then, can assign agiven vibrating element to individual signal processing, individualsynthesizer voice, or both in combination. This permits a basicconventual instrument structure (such as a guitar, violin, steel guitar,koto, or mbira) and essentially conventional playing techniques tocontrol an unprecedented rich range of sounds. (Even more recently,Roland has since announced a cable fan-out product allowing their VG-1multi-channel signal processor to operate simultaneously with theirGP-10 and related guitar to MIDI interface products.) Furtherenhancements are also possible. For example:

[0543] In practice it may be desirable to have a different number ofsignal processors than vibrating elements. For example:

[0544] in generalized steel guitars, only a few strings at a time mayactually be candidates for pitch shifting

[0545] in spatial-spectral (panning, chorusing, etc.) animation, theactual number of animation channels need not match the number ofvibrating elements

[0546] in a highly functional system, several signal processors may beused in parallel for one or more vibrating elements.

[0547]FIG. 53 shows partial mix-downs of vibrating element signals fedto a number of signal processors 5005.1-5005.n and straight-throughpaths 5007.1-5007.m en route to subsequent mix-down. The (pre-signalprocessing) input mixer 5006 is used to route and/or mix variousmulti-channel transducer signals to a structured and controllablemulti-channel mix. Several variations of this arrangement are suggestedby FIG. 53:

[0548] there can be zero, one or more straight paths 5007.1-5007.m.

[0549] there are a total of at least three signal processors5005.1-5005.n and/or straight paths 5007.1-5007.m involved.

[0550] there is a minimum of one signal processor (or else the twomixers 5003 and 5006 functionally collapse into one, functionallyresulting in the arrangement of FIG. 50).

[0551] although the interconnection details for connecting thesynthesizer interface are shown, the synthesizer interface need not beincluded.

[0552]FIG. 54 shows another arrangement wherein a switching matrix 5008is used in place of the input mixer 5008 to select which individualvibrating element 5001.1-5001.n signal is assigned to which signalprocessor 5005.1-5005.k. This arrangement is particularly relevant tothe generalized retuning of certain vibrational elements, as in adaptedpedal steel guitar and adapted sitar described earlier.

[0553] The invention also provides for several signal processors to bepooled and used in various parallel, series, or other topologicalinterconnections serving one or more vibrating elements. FIG. 55 shows amore flexible example method for providing signal processors withvibrating elements' signals and other signal processor outputs viaswitch matrix, and additional partial mix-downs by replacing said switchmatrix 5008 with an input mixer 5006.

[0554] The invention provides for any of the above systems to beintegrated together into a common system sharing a common configurationpreset storage and recall facility. FIG. 56 shows configuration controlof signal processors, mixers, any switch matrix, and synthesizerinterfaces via logic circuitry and/or microprocessing. As illustrated inFIG. 56, this is simply a matter of putting all or some combination ofthe mixers (5003 and/or 5006), switch matrices 5008, signal processors5005.1-5005.k, and/or synthesizer interfaces, as relevant, under thecontrol of logic circuitry and/or microprocessors 5009 which can providesuch preset storage and recall functions.

[0555] By combining the multi-channel signal handling with excitation,not only can individual vibrating elements be assigned to various signalprocessing and synthesizer controlling roles, but also individualvibrating elements can now be assigned feedback modes where selectedvibrating elements can sustain vibration as if they were bowed, in anelectric-guitar feedback arrangement, etc. Further, through use ofadditional switching, signal processing can be added to the feedbackloop as discussed earlier, but on an individual vibrational elementbasis. Finally, since feedback arrangements tend to emphasize higherharmonics of vibration, and the dynamics of the relative levels of theharmonic mix can be varied dramatically by touching elements or varyingfeedback characteristics (via signal processing in the excitationfeedback loop), the invention provides for control signal extraction tobe expanded to respond to details of the overtone content as discussedlater.

[0556]FIG. 57 shows a very general combined environment formulti-channel signal processing, mixing, excitation, and program controlof overall configuration. In FIG. 57 the general combined environmentincorporates a plurality of separate feedback loops for each vibratingelement, each loop featuring a loop signal processor 5021.1-5021.n(which here could be as simple as a level control) which may becontrolled by control signals as provided for in the invention. Manypossible variations of this approach which may omit or simplify any ofthe elements shown in FIG. 57 can be realized; for example the signalprocessors 5005.1-5005.1 k and 5021.1-5021.n may be pooled together intheir association with mixers 5006 and/or switches 5008 with said mixers5006 and/or switches 5008 sending signals to the excitation driveamplifiers 5022.1-5022.n.

[0557] 7.2 Audio Signal Processing

[0558] Many of the audio signal processing elements cited as 125 (FIG.1), 2211 (FIGS. 25-26), 5005.1-5005.n/k (FIGS. 51-57), and elsewhere inthis document can be adequately realized by any number of the standardmulti-function MIDI-controlled signal processing modules such as theRoland model RSP-550, Boss model SE-70, Ensoniq model DP/4, ART modelSGE Mach-II, Digitech model GSP21, etc. In the invention, these signalprocessing functions are preferably an integrated component within alarger-scale hardware and software construct rather than anoff-the-shelf module. The invention provides not only for theincorporation of these into the signal routing, processing, andsynthesis entity 120 (FIGS. 1-2) in the ways described below but alsofor additional audio signal processing methods which are notcommercially available and which are described below.

[0559] 7.2.1 Spatially Distributed Timbre Construction

[0560] Because of the extensive biaural capabilities of human hearing,stereo and other multichannel sound fields can be used to create anumber of musically useful timbral construction ranging from the subtleto the powerful and the beautiful to the bombastic.

[0561] Examples of this, commonly found, are stereo-output chorus,stereo-output flangers, stereo-output reverb, stereo-output echos, etc.;but the spatial construction of timbres may be carried far beyond thesesimple and now commonplace effects. The following discussion explainssome example techniques; the role and value of these techniques aredeveloped further in subsequent material following that below.

[0562] 7.2.1.1 Cross-channel modulated delay

[0563] The invention provides for methods to enhance, and to moresignificantly increase the depth of, a stereo signal set source whosecomponents have similar but slightly different timbres, particularly ifthe timbres are time-varying. Examples of such stereo signal set sourcesinclude the stereo outputs of traditional choruses, flangers, reverbs,etc., a pair of signal distortion elements with differentcharacteristics, two harmonized synthesizer voices or pitch-shifteroutputs, the separate outputs of a single two-oscillator synthesizervoice, etc. FIG. 58 shows a stereo-input, stereo output configuration oftwo monaural flange and/or chorus elements wherein the unaltered signalof each input channel is combined with a delay-modulated signal from theopposite channel. Each input 5801, 5802 is presented to a dedicateddelay-modulation element 5809, 5810 and a dedicated output channelsummer 5831, 5832. Internally each delay-modulation element 5809, 5810consists of a variable delay implemented in this example by changing theclock speed of a clocked delay line 5803, 5804 by means of a variableclock oscillator 5805, 5806. The speed of the clocking oscillator iscontrolled by a variable low-speed modulating “sweep” oscillator 5807,5808. Since the sweep oscillator's output is periodic, it is possible toimplement the clocked delay line 5803, 5804 not only by changing thesample rate but by loading fixed-rate samples into a ring buffer andreading out of the ring buffer at a rate set by the variable clock 5805,5806.

[0564] Other implementations of delay-modulation elements 5809, 5810 arealso possible as known to those familiar with the art. The outputs 5821,5822, respectively, of each delay-modulation element 5809, 5810respectively, are directed to the summing elements 5832, 5833,respectively associated with the opposite unaltered input channel 5802,5801, respectively, producing stereo outputs 5832, 5831. The delayedsignals 5821, 5822, may be summed at the summing elements 5832, 5833 inadditive, subtractive or other phase-shafted or phase-dispersedrelationships. In a preferred implementation the two sweep oscillators5807, 5808 operate at slightly different sweep frequencies and arefree-running (unsynchronized) if possible.

[0565] (It is noted that a similar, restricted version of this has sincebeen incorporated as one of the effector modes, namely “cross-overchorus”, of the Korg model X5DR synthesizer module. In the Korgimplementation, however, the two sweep oscillators 5807, 5808 have beenreplaced by a single sweep oscillator with two phase-locked quadrature,i.e., 90-degree phase difference, outputs.

[0566] It is noted the above arrangement may naturally be extendedbeyond stereo to accommodate additional input and/or output channels.The most general implementation would have N inputs, M outputs, N−1variable speed swept delays, and M summers with N inputs summed withadjustable gains and/or phase relationships; simplifications of courseare possible. One example application would include M-speaker (i.e., M=4for quadraphonic) amplification. Another example application with M=2for stereo and N>2 similar signal sources would build an enhancedversion of the sonic effect.

[0567] In the above it is noted that when pluralities of elements (forexample, spatializer and distortion elements) are cited, the elements inthe plurality need not be identical in their type and/or parameterizedsettings. Further, various parameters of each of the elements(modulation speed, modulation depths, relative amplitudes in audiomixes, etc.) may be advantageously controlled in real-time by controlsignals for expression (from instrument entities, foot controllers,etc.), further correlation with the signal source (for example, usingenvelope extraction control signals) or further levels of animatedenhancement (employing additional sweep oscillators, envelopegenerators, etc.).

[0568] 7.2.1.2 Multi-Level Stereo Chorused Distortion of MonauralSources

[0569] The invention provides for creating a similar-signal stereosignal set from two distortion sources and presenting it tocross-channel modulated delay to synergistically transform a relativelyspectrally dull signal, particularly a time varying one, into a veryrich powerful sound. FIG. 59 illustrates a combination of a spatializedeffect, two distortion elements, and a stereo (N=M=2) cross-channelmodulated delay. In this example arrangement, the input signal 5900,which may be from a group pickup as shown, individual vibrating elementpickup, audio signal synthesis element, microphone, etc., is applied toa stereo-output spatializing effect 5901 such as a stereo output chorus,flanger, etc. The resulting stereo signals 5902, 5903 are applied todistortion elements 5904, 5905. Fluctuations in the waveshape of theapplied stereo signals 5902, 5903 cause significant dynamic timbralshifts from the distortion elements 5904, 5905; such fluctuations in theapplied stereo signal waveshape may originate from the original signal5900, from any one or more of amplitude animation, phase animation,delay accumulation, dynamic waveshaping processes, etc. in thespatializer 5901, or both in combination. The distortion elements 5904,5905 produce a rich similar-signal stereo signal set 5906, 5907 which isthen presented to a N=M=2 cross-channel modulated delay, resulting instereo output signals 5909 a, 5909 b.

[0570] For a single input channel 5900, the invention provides for theexpansion of such an arrangement to include additional processes tobuild an enhanced version of the sonic effect. For example, an N-outputversion of the spatializer 5901 (which may, for example, be implementedinternally by two or more simpler spatializers in parallel,hierarchical, or other interconnection topologies) can be used inconjunction with N distortion elements in an N-input (N>2) M -outputcross-channel modulated delay replacing 5908.

[0571] For multi-channel signal sources, the invention provides for eachsignal to be handled by a dedicated spatializer and several possiblesubsequent processing arrangements. As one example, assuming K inputchannels, selected outputs of each of the K spatializers may be mixedand presented to N (N being two or more) distortion elements which inturn are presented to an N input, M output cross-channel modulated delayreplacing 5908. In another example, no pre-distortion mixing is used butrather each spatializer is provided with its own collection of two ormore distortion elements; the collection of all outputs of these, whichare of number J not equal to N, may be matrix-mixed to form N mixedoutputs which are applied to an N input, M output cross-channelmodulated delay replacing 5908. In another example, no pre-distortionmixing is used but rather each spatializer is provided with its owncollection of two or more distortion elements; the collection of alloutputs of these, which are of number N, may be directly applied to an Ninput, M output cross-channel modulated delay replacing 5908. Otherarrangements similar in form and spirit are clearly possible.

[0572] In the above it is noted that when pluralities of elements (forexample, spatializer and distortion elements) are cited, the elements inthe plurality need not be identical in their type and/or parameterizedsettings. Further, various parameters of each of the elements(modulation speed, modulation depths, relative amplitudes in audiomixes, distortion parameters, etc.) may be advantageously controlled inreal-time by control signals for expression (from instrument entities,foot controllers, etc.), further correlation with the signal source (forexample, using envelope extraction control signals) or further levels ofanimated enhancement (employing additional sweep oscillators, envelopegenerators, etc.).

[0573] 7.2.1.3 Location Modulation

[0574] Location modulation has been commercially available in the formof “auto-panning” where an audio source is periodically panned back andforth between two stereo outputs. The invention provides for limitingperiodic auto-panning of monaural sources sounding in isolation to betypically most effective when the degree of panning is limited and themodulation rate is low (as extreme settings of modulation depth andspeed are typically not as widely musically useful). Under theseconditions in a stereo sound field a signal source takes on an animatedcharacter but yet is not so blatantly spectrally modified as it is inchorus and flanging effects. The invention also provides for widerranges of depth and speed to be used in the context of multi-channelauto-panning, discussed next, and layered signal processing discussedbelow and already touched upon in the discussion associated with FIG.39.

[0575] The invention provides for multi-channel versions ofauto-panning. In layered signal processing, such as that discussed inthe context of FIG. 39, auto-panning contributions work best within theinvention if modulation sweep oscillators operate at different(typically only slightly different) modulation speeds. In the context ofmulti-channel signals provided by individual vibrating elements of aninstrument entity 100, this unsynchronized configuration of modulationsweep oscillators can create inhomogeneous “bunching” effects when manymodulation sweep oscillators take on identical or nearly-identicalvalues in extreme ranges of the modulation sweep.

[0576] The invention provides for a much more homogeneous method formulti-channel periodic-sweep auto-panning, namely that of arranging thesignal pan images in a phase-staggered constellation swept by a singlemodulating sweep oscillator. A simple example is that of stereocross-panning where two input signals pan between stereo speakers insynchronized complementary directions. Another example is that ofstaggering the phases of a multiple phase output modulating sweeposcillator in some preassigned arrangement, such as offset from eachother by a common phase-offset value. This may be used to pan the soundsfrom each individual vibrating element so that the individual pannedsound images follow one another between two speakers. Similar methodscan be used if there are more outputs (for example, quadraphonic,hexaphonic, octaphonic etc. speaker installations aligned in a plane orin 3 dimensions); here N input, M output mixers can be controlled by oneor more single or multiple-phase output modulating sweep oscillators.

[0577] Control-signal invoked transient “one-shot” panning effects mayalso be obtained from commercial mixer products that feature a fade-timetransient between pre-programmed amplitude settings (such as the YahamaDMP series and Sound Sculpture Switchblade series). The inventionprovides for such transient effects to be used as a compositionalelement in music or a metaphorical or semiotic element in audio and/oraudio-visual aspects of performance. In particular limited-durationpanning trajectories of arbitrary nature, each affiliated with one ofseveral individual sound sources, may be made to simultaneously and/orsequentially follow a predefined relative dynamical pattern. This can beused as a contrapuntal element in melody or abstract musical forms. Itcan also be used to create plot events in a composition or performance,such as in a musical composition, dance composition, or play concerningor involving the spatial interaction of bird sounds.

[0578] 7.2.1.4 Other Spatially Distributed Timbre Methods

[0579] Several other aspects of the invention to be presented below inother contexts also may be used to create spatially-distributed timbralrealizations; their use as general audio signal processing elements 129a in this fashion is provided for as part of the invention.

[0580] One aspect of the invention which may be used forspatially-distributed timbral realizations is: the two-input ormultiple-input versions of the octave cross-product chain describedlater on in the context of audio signal synthesis waveshaping. Asdescribed there, this technique results in a number of parallel signaloutputs with widely differing spectral contents and spectral animationfeatures, and the animation features slow to a halt when all fundamentaland overtone frequencies of the two input signals are brought into fixedinteger and small integer-ratio multiplicative relationships. Theaforementioned characteristics of the multiple outputs lend themselvesto spatially-distributed timbral realizations since mixing of theoutputs can partition the frequency content and animation featuresdifferently between final mix-down outputs. The invention provides forthis method to be used as a signal processing technique. In one exampleusage, a pitch-shifter, swept variable delay, etc. is used to constructa derivative frequency and/or phase shifted signal (the characteristicsof which may be controlled by control signals for expression) from anoriginal signal. The original and derivative signals are then fed intothe octave cross-product chain to produce often spectacularspatially-distributed timbral realizations.

[0581] Another aspect of the invention which may be used forspatially-distributed timbral realizations is multi-channel waveshapingwhere a signal source is provided to a plurality of waveshapers each ofwhich may be controlled by control signals. Each waveshaper output maythen have differing frequency content and animation features which thuslend themselves to spatially-distributed timbral realizations in amulti-channel (stereo, quadraphonic, etc.) partition or mix-down. Ofparticular interest is the use of hysteretic waveshaping, describedlater, which creates a wide range of spectral differences as the inputwaveform and/or hysteresis parameters change over time.

[0582] Another aspect of the invention which may be used forspatially-distributed timbral realizations is the use of later describedlayered audio signal processing methods. The invention does this byproviding for each audio signal processing layer to be allocated adifferent proportion to each final mix-down output channel. Theseallocated mix proportions may be varied over time by control signals.

[0583] 7.2.2 Multi-channel Audio Signal Handling

[0584] The invention provides for flexible homogeneous and inhomogeneoussignal processing of multi-channel audio sources. Such multi-channelaudio sources may for example include, referring to FIG. 1-2, a multiplevibrating element instrument entity 100, multiple instances of audiosignal synthesis elements 129 a with single or multiple output-channelsaudio signal synthesis elements.

[0585] Several signal processing methods involving multi-channel signalsources have already been discussed thus far, particularly those in theprevious few sub-sections. The invention further explicitly provides fordedicated, shared, or combined arrangements for audio signal processingelements within the signal routing, processing, and synthesis entities100 as shown in FIGS. 53-57. In particular, each signal from amulti-channel source may be handled differently with at least some ofthese signals processed by one or more audio signal processing elements129 a of identical, similar, or differing function. Conceptually, themost flexible of these are embodiments where input mixing 5006 is usedto share the inputs and outputs of a pooled collection of signalprocessors as in FIGS. 55-57. The audio signal processing elements 129 amay be any one or more of conventional signal processing functionsprovided by commercial products (chorus, flange, reverb, distortion,delay, filtering, equalization, etc., individually or in combination) aswell as any one or more of the invention's novel audio signal processingmethods described thus far and below.

[0586] 7.2.3 Bass Note Derivation

[0587] The invention provides for the derivation of bass notes fromsignal sources. This is particularly relevant in the invention wheresignals from selected vibrating elements are used to create bass notes.The created bass notes may be heard in parallel with the original pitchof the signal (each pitch may be subject to different signal processing)or in replacement of it.

[0588] In many cases this completely eliminates the need for bassaccompaniment in a performance situation at the potential expense ofmelodic freedom of the bass line.

[0589] The invention provides for at least three methods of bass notederivation which may be used individually or in combination.

[0590] One of these methods is the use of control signal extraction toderive note events to run a bass note audio synthesis element (forexample, a conventional audio synthesizer module transposed down one ormore octaves or other large interval). If the bass interval is notalways to be fixed, pre-programmed note transpositions reflectingdesired harmony and/or player-controlled changes in pitch-shift intervalmay be used individually or in combination. This audio synthesizermethod allows a wide range of sounds to be used but can be limited inhow the bass note expression can be controlled from the original signalsource. One solution to this provided for by the invention is the use ofovertone parameter tracking in the control signal extraction; theseadditional parameters may be used to shape the synthesized sound thoughvarying parameters in the synthesis processes and/or by varyingsubsequence signal processing parameters.

[0591] Another of the methods is through the use of conventional pitchshifters. If the bass interval is not always to be fixed, so called“intelligent-harmony” pitch shifters (such as the Digitech model IP-33B)and or player-controlled changes in pitch-shift interval may be usedindividually or in combination. The use of pitch-shifting allows fornuances of the original signal source to be carried through but maysuffer from delayed response, glitch, phasing, “Darth Varder,” or otherundesirable or limiting artifacts.

[0592] Yet another method, should the bass interval always be related tothe source pitch by octaves, the invention provides for an adaptation ofthe novel octave divide method used in the Boss OC-2 “Octaver” pedal.Although this technology does have glitching and monophonic limitationsas described below, it works very well in responding to amplitudeenvelope attributes of the signal source. As is evident from thepublicly available published service note schematic and usage of thedevice, each octave signal is created by frequency dividing the originalsignal (for example by means of a toggle flip-flop), scaling itsamplitude by the instantaneous amplitude of the source signal (forexample, through use of an envelope follower and a gain-control method),and combining this with a bit of the original signal to create a richerresulting overtone result. The unit suffers from the fact thatharmonically rich signals often confuse the frequency dividers resultingin a very glitchy bass signal. Further, the method is monophonic; theplaying of two notes at once processes only one bass signal, and usuallyan unusably unstable one. The invention provides for the glitch-freeadaptation of the OC-2 technology to multiple vibrating elementinstruments by dedicating a specific low-pass filter and an allocated(or allocatable) OC-2 divider or divider chain to each selectedvibrating element. In particular, the incoming individual vibratingelement signal is low-pass filtered to greatly attenuate frequenciesabove the maximal fundamental frequency to be recognized by thearrangement (this maximal value may, in some circumstance be high enoughto support unfretted string “chime” harmonics and the like).

[0593] The combination of applying each instance OC-2 technology to asingle vibrating element together with a highly emphasized fundamentalfrequency eliminates the glitching and monophonic limitations. Theinvention provides for a plurality of the described OC-2/filterarrangements, numbering for example three for a guitar, to be allocatedto specific vibrating elements (fixed by design, selectable via storedprogram control, etc.). Further, the invention provides for the use ofthis technology should bass notes need to be non-octave in relation tothe original signal: the nearest octave note can be generated by theOC-2/filter approach and an allocated pitch shifter may be used to makerelatively smaller pitch changes, recognizing that smaller shiftintervals tend to have less artifacts.

[0594] 7.2.4 Layered Audio Signal Processing

[0595] The invention provides for the layering of multiple audio signalprocessing paths driven from one or more shared sources and partitionedor mixed down to two or more output channels. Because this may be viewedas a superposition of several signal processing paths, this will bereferred to as “layered audio signal processing.” One example of thishas already been presented in the discussion relating to FIG. 39; hereeach layer is responsible for emulating a separate sympathetic stringeffect. As each layer is in the FIG. 39 example essentially identical,the layers may be called “homogeneous.” (Some examples of homogenoussignal processing have since been devised, for example the “PentaChorus”preset of the Roland RSP-550 signal processing module.) In contrast tohomogeneous layered signal processing, FIG. 60 illustrates separableexamples of inhomogeneous layered signal processing. The examples ofFIG. 60 may be used as shown, with selected omissions, or as an archtypefor similar constructions as provided for in the invention. In oneexample a group pickup signal 6001 is applied to a distortion element6011 and a compression element 6013. Each of the output's signals 6104,6005 as well as the original signal are provided to separatespatializers 6012-6015 (for example, chorus, flange, reverb, etc.) whichare then mixed down into a stereo signal 6021 a, 6021 b by an outputmixer 6020. The invention provides for the substitution of any of theelements 6011, 6013, 6012-6015 with other types of audio signalprocessing elements as well as inclusion of additional layers. Asexpansion of the example, individual vibrating element signals fromvibrating elements sharing the aforementioned group pickup can also beused to create additional layers. For example, an individual bass stringsignal 6002 (for example, the 5th and or 6th string of a guitar) may, inparallel, be processed by a pitch-shifter, OC-2/filter, etc. 6016 tocreate a bass note pitch signal 6006 which, in turn, may be presented toa separate spatializer 6017. Further, another individual string signal6003 can be processed in a similar fashion but replacing thepitch-shifter with an emphasis effect for emphasizing a particularmelody or note in a chord.

[0596] Because of the larger number of sonic sources that can bestatically distributed in the sound field, the invention provides forthe use of location modulation with a wider range of permissiblemodulation rates and modulation depths as extremal location modulationbehavior is only part of the overall spatial sonic structure.

[0597] The invention also provides for the use of layered audio signalprocessing in the creation of spatially-distributed timbralrealizations. One example of this would be providing a dedicated stereochorus to each of the six individual string signals of a guitar as wellas a seventh stereo chorus to the group pickup signal, setting eachchorus sweep rate slightly differently and summing the seven stereooutputs into a single stereo mix; this is in fact an example adaptationof the principals illustrated in FIG. 60. Another example is that of thecross-product octave chain to be described later.

[0598] The invention provides for the use of waveshaping techniques,particularly those which can be varied in real-time by control signalsand/or hysteretic waveshaping techniques, as signal processing elements.The invention also provides in general for the separate and/orcoordinated control of parameters involved at each audio signalprocessing layer by means of general control signals.

[0599] 7.2.5 Envelope-controlled time and pitch modulation

[0600] The invention provides for the modulation of the delay time of avariable delay line by a control signal corresponding to the amplitudeenvelope of the delayed signal or an associated signal. This causes atape-recorder speed instability effect correlated to the transientcharacter of the reference signal amplitude envelope; more precisely thepitch changes with the time derivative of the amplitude envelope. Theinvention also provides for the substitution of a variable pitch shiftercontrolled by the time derivative of the same control signal; thisarrangement produces roughly the same effect. In either implementationthe control signal may be first warped by an emphasis non-linearity,control signal delay, and/or other processing functions. The result canbe used in soloing as a climactic effect or in moderation for atransient enhancement. The invention also provides for envelope controlof pitch-shifting without time-differentiating the control signal.

[0601] 7.2.6 Resonant distorting delays

[0602] The invention provides for the sitar-like sympathetic/buzzemulation utilizing short high-resonant delays as described inassociation with FIG. 39 to be used as a more general signal processingelement. This is particularly useful if parameters of the configuration,such as degrees of resonance, degrees of clipping, and modulationdepths, can be varied in realtime by control signals.

[0603] 7.2.7 Hysteretic waveshaping and distortion

[0604] Hysteresis occurs to some extent in overdriven tube amplifieroutput transformers due to the natural hysteretic properties of thematerials used to make the transformer core. Hysteresis effects inwaveform distortion can create valuable amplitude-varying . . . . Theinvention provides for generalized models of hysteresis to be used as awaveshaping technique, and as such a signal processing technique, withparameters of the hysteresis action variable in real-time via controlsignals.

[0605] Traditional hysteresis curves for transformers, gears,pseudo-elastic deformation, etc. are well known (see for example[Visintin]. FIG. 61 illustrates an example of a generalized hysteresismodel construction as provided for by the invention. The input/outputgraph shows example symmetric curves that are linear 6102, superlinear6103, and sub-linear 6104 along with the time/amplitude oscillograph ofan example applied waveform 6110.

[0606] Other types of symmetric or non-symmetric non-linearities mayalso be used. A time-derivative operation on the applied signal waveform6110 followed by sign detection reveals whether the applied signalwaveform is at any instant increasing or decreasing. As an example, theapplied signal waveform 6110 would be applied to one non-linear warpingfunction such as 6103, 6104 when increasing and the other whendecreasing, resulting in the waveform made of segments 6113, 6114 ratherthan the waveform 6112 that would have been created by the linear curve6102. In order to allow the applied input signal to vary in amplitudeand still maintain continuity of the waveform, the invention providesfor the warping non-linearities to be themselves adaptively scaled orotherwise altered based on amplitude information from the current andprevious direction reversals, moving average of waveform area orwaveform power, etc. The invention provides for aspects of thehysteresis process, such as curve shapes and degrees of dependency onwaveform history, to be varied in real-time by control parameters.

[0607] Hysteretic waveshaping can be of use in layered audio signalprocessing and spatially-distributed timbral realizations which havebeen described above.

[0608] 7.3 Audio Signal Synthesis

[0609] Referring to FIG. 1-2, the invention provides for the inclusionof audio signal processing elements 125 with the signal routing,processing, and synthesis entity 120. These may include conventionalMIDI synthesizers, analog synthesis elements, or other technologiesparticular to the invention such as cross-product octave chains,hysteretic waveshaping, vowel sound synthesis, etc. The inventionprovides for as many parameters as possible to be variable in real-timeby means of standardized control signal formats. The invention alsoprovides for the synthesis of vibrating element feedback soundscontrolled by control signals and pitch sampling as may be adapted fromthe Boss DF-2 “Distortion/Feedback” product.

[0610] 7.3.1 Spatially Distributed Timbre Construction

[0611] It is possible to create spatially distributed timbrerealizations as part of the audio synthesis process as well as bysubsequent signal processing (cross-channel modulated delay, multi-layerchorused stereo distortion, phased multi-signal constellation locationmodulation, etc.) as described earlier. The invention provides forspatially distributed timbre realizations within synthesis by a varietyof methods. One method, found in many commercial synthesizer modules(such as the Korg M3-R, Korg X5DR, and Kawia K4-r, for example), is forthe synthesizer voices themselves to involve multiple paralleloscillators and/or sample-players delivered in the stereo or other multioutput form. This sub-section discusses two other methods provided forby the invention.

[0612] 7.3.1.1 Cross-product octave chain

[0613] The many times aforementioned cross-product octave chain involvestwo or more octave divider chains whose corresponding outputs aremultiplied together, with all resulting outputs summed together by amultiple output mix-down mixer. The cross-product technique results in anumber of parallel signal outputs with widely differing spectralcontents and spectral animation features, and the animation featuresslow to a halt when all fundamental and overtone frequencies of the twoinput signals are brought into fixed integer and small integer-ratiomultiplicative relationships. The aforementioned characteristics of themultiple outputs lend themselves to spatially-distributed timbralrealizations since mixing of the outputs can partition the frequencycontent and animation features differently between the final mix-downoutputs. The invention provides for the incorporation of cross-productoctave chains in audio single synthesis.

[0614]FIG. 62 shows an example implementation of a cross-product octavechain particularly suited to low cost implementation with logic chips orsimple DSP program loops. Two input signals 6201 a, 6201 b are appliedto optional comparators 6202 which convert the applied signals intotwo-value waveforms. These are applied to a chain of octave droppingelements 6203 which here can be implemented in isolation as a chain oftoggle flip-flops and in aggregate as a binary counter. The depth of thechain can include many levels with three to six a useful number oflevels. Each of the resulting output signals at corresponding levels aremultiplied together by multipliers 6204. Signal analysis of the truthtable for various logical operations show that for applied square wavesan EXCLUSIVE-OR operation acts exactly as a unity-gain multiplication(while an AND function acts as a unity-gain multiplication added tohalf-amplitude versions of the two applied square waves), so themultipliers 6204 may be realized by EXCLUSIVE-OR operations. Thisamounts to less than $1 US worth of chips in hardware and a small amountof code in software. All outputs are provided to an output mixer 6205which produces at least stereo outputs 6206 a, 6206 b as well aspotentially other mix-down outputs 6207.1-6207.n and which in apreferred embodiment may be adjusted in real-time by control signals.

[0615] The invention provides for alternate implementations, fro exampleomitting the comparators 6202, implementing the octave drop functions6203 with pitch shifters or OC-2/filter technology, and/or implementingthe multipliers 6204 with VCAs or 4-quadrant multiplier operations. Theinvention also provides for expansions to include more than two octavechains.

[0616] In the context of audio signal synthesis, the applied signals6201 a, 6201 b may be generated by two oscillators within a singlesynthesizer voice; these oscillators may be relatively tuned in unison,octaves, or near-consonant intervals for basic operation, and one of theoscillators may be continuously swept through a range of pitches tocreate huge audio displays of pleasing spectral complexity.

[0617] 7.3.1.2 Multi-channel Waveshaping

[0618] The invention also provides for spatially distributed timbrerealizations through use of parallel or complementary modulations of aplurality of waveshaping operations by control signals. The outputs ofthe plurality of waveshapers are then mixed into a stereo ormultichannel output mix.

[0619] 7.4 Control Signal Routing

[0620] The invention provides for extensive control capabilities and assuch requires sophisticated control routing, processing, and storedprogram organization. The capabilities for this provided by theinvention are described in the following sub-sections. To illustrateessential capabilities the discussion below is stated in terms ofcommonly appreciated MIDI messages and conventions, but the inventionprovides for these same capabilities to apply to other signal formats indigital, analog, contact closure, entirely software, etc. or anycombination.

[0621] 7.4.1 General control signal switching and merging

[0622] Referring to FIGS. 1-2, input signals directed to control routingand merging include the control outputs from the instrument entities 100(including foot controllers), control signal extraction elements 128 a,control signal processing elements 123, and control signal synthesiselements 129 b. Still referring to FIGS. 1-2, output signals directedfrom control routing and merging include control inputs to instrumententities 100, all elements within the signal routing, processing, andsynthesis entity 120 (probably excluding the power supply 121), as wellas any external lighting and/or special effects control systems.

[0623] Using MIDI messages and conventions as a model, control signalsmay be carried through cables and subsystems in combinations ofmultiplexed formats (the sixteen MIDI channels plus the variety ofmessage types) and space-division formats (multiple MIDI cables). In theMIDI context the invention provides for control signal routing at theMIDI port (i.e., MIDI cable) level, the MIDI channel level, and themessage index (MIDI note numbers, MIDI Continuous Controller numbers,etc.) level. This same hierarchy of routing capabilities would alsoapply to non-MIDI control signal equivalents. The invention alsoprovides for the processing of control signals at any of these levels.

[0624] The MidiTemp MIDI processor products are by far the mostcomprehensive commercial products known at this writing; they providefull-capability port level and channel level routing but only verylimited capabilities at the message index level. Further, the inventionprovides for control switching and merging functions to preferably be anintegrated component within a larger-scale hardware and softwareconstruct rather than an off-the-shelf module.

[0625] To aid in using control signals throughout the system, theinvention also provides for visual indicators of control message value,such LED bar-graphs which may be accessed through control signalrouting.

[0626] 7.4.2 Multi-channel Control Signal and Stored Program Handlingand Organization

[0627] The invention provides for a flexible control and configurationhierarchy for signal routing, processing, and synthesis entities.

[0628]FIG. 63 illustrates an example flexible control and configurationhierarchy for control signal and stored program handling andorganization. In the Figure, all “program” entities (with the potentialexception of the configuration program 6310) are stored programs whichcan be recalled up and swapped under the command of control signals(such as MIDI Program Change commands). Each stored program can in turncase the recall and/or swapping of other stored programs in accordancewith those arrowed lines between pairs of programs as shown in FIG. 63.

[0629] Referring to FIG. 63, input control signals 6300.1-6300.n frominstrument entities, foot controllers, etc. are applied to a controlsignal routing and control signal processing environment 6301. Thecontrol signal routing and control signal processing environment 6301may internally be decomposed into separate control signal routing andcontrol signal processing elements or instead be integrated together ina common realization (as is common on many MIDI signal routing andhandling products). The control signal routing and control signalprocessing environment 6301 then distributes output control signals6302.1-6302.m throughout the rest of the system, specifically allclasses of elements depicted in FIG. 63, potentially including controlaspects of itself. Among the elements receiving control signals is aconfiguration program 6310 which potentially provides simply abackground environment defining specific ports, safeguards, and anycommon control distribution frameworks (point to point, broadcast,daisy-chain) should this be necessary. A variety of configurationprograms 6310 may be made available for varying operational modes (forexample stand-alone operation, ganged operation with one or more othersignal routing, processing and synthesis entities, backup modes,diagnostic modes, etc. A more significant receptor of the output controlsignals 6302.1-6302.m is the master program 6320. This stored programcan change the configuration of the control signal routing and controlsignal processing environment 6301 as well as the choice of any of thesubsystem stored control programs 6331-6337 affiliated with sequencing,audio, lighting, etc. These control programs in turn may change theconfiguration of the control signal routing and control signalprocessing environment 6301 (specifically 6331, 6332, 6334, 6335 areshown with this capability in FIG. 63) as well as the choice of any ofthe stored programs in the clusters of specific subsystem elementsrepresented by 6342-6347 as well as the potentially external lightingsubsystem element cluster 6348; specifically these stored programcommand paths are as indicated by the arrows in FIG. 63. Further, thesequencer control program calls up and potentially immediately initiatesreal-time control signal sequences 6341 some of which may also changethe configuration of the control signal routing and control signalprocessing environment 6301 as indicated by the arrowed line in FIG. 63.

[0630] It is understood that FIG. 63 serves as an illustrative exampleand that the invention provides for other organizational structures ofthis flavor and spirit.

[0631] 7.5 Control Signal Processing

[0632] The invention provides for control signal processing to beincluded so as to add extensive valuable control capabilities. Forconvenience these control signal processing operations are described interms of MIDI; the invention provides for these capabilities in othercontrol signal formats as well.

[0633] Monodic operations:

[0634] intelligent harmony (note by note remapping, individually or inranges of arbitrary size)

[0635] note-number to MIDI Continuous Controller values

[0636] note-velocity to MIDI Continuous Controller values

[0637] MIDI Continuous Controller values to note number messages

[0638] MIDI Continuous Controller value transformed by fixed scaling andoffset values

[0639] MIDI Continuous Controller values (0-127) remapped to arbitrarymappings by point, by line segment, or by fitted curve segment

[0640] MIDI Continuous Controller complementary value transformation(i.e., if received value is “x”, transmitted value is “127-x”)

[0641] message delay

[0642] message value threshold tests resulting in the issuances of newmessages

[0643] message value threshold tests resulting in selected routingchoices for the received message.

[0644] Polyadic operations:

[0645] multiplication of MIDI Continuous Controller values

[0646] scaling and offset of MIDI Continuous Controller valuescontrolled by other MIDI Continuous Controller values

[0647] MIDI Continuous Controller to Note number and Note velocity

[0648] sequence detection in a received series of MIDI messages,potentially within a defined time window, resulting in a new issuedmessage

[0649] 7.6 Control Signal Extraction

[0650] The invention provides for the extraction and derivation ofcontrol signals from audio and video signals as described below

[0651] 7.6.1 Audio signal to MIDI Note Event

[0652] The invention provides for the conversion of received audiosignals into note events as is standardly done in products such as theRoland GP-10, GM-70, and CP-40. The invention also provides for moreadvanced extractions and derivations as explained below.

[0653] 7.6.1.1 Envelope Tracking to MIDI

[0654] The above conversions of received audio signals into note eventsas is standardly done in products such as the Roland GP-10, GM-70, andCP-40 have been limited to channel allocation, note number and notevelocity. The invention provides for the real-time extraction ofamplitude envelope information and its conversion to control signals.For example, the amplitude envelope may be used to control a signalprocessor or signal pan location. Because the amplitude envelope fallsoff in a typically exponential way over time while most controlstructures expect linear variation, the invention provides for one ormore possible warpings of the envelope signal, such as logarithm orpiece-wise linear constructs. Further, the invention also provides forhigh-pass, band-pass emphasis/notching, and low-pass filtering prior toparameter extraction so as to limit unwanted influence of audio signaltransients at the initial execution of a vibrating element or audiosynthesized note.

[0655] 7.6.1.2 Control Signal Extract from Vibrating Element Overtones

[0656] The use of pitch-detecting interfaces for converting the pitchedvibrations of individual vibrating elements into control signals for usewith synthesizers or other musically-oriented signal processing has beenin use for many years, particularly since shortly after the invention ofthe MIDI standard for electronic instrument control. However, suchpitch-detecting interfaces have derived only the fundamental frequencyand overall amplitude of the pitched vibrations of individual vibratingelements of an instrument. The use of filter banks for determining theenergy in course frequency bands for the purposes of controllingmusically-oriented signal processing (i.e., the so-called vocoder”) isalso known. However, the practice of determining the scale-accuratepitches and amplitudes of individual overtones for the purposes ofcontrolling synthesizers or other musically-oriented signal processingis currently not known.

[0657] Current synthesizer interfaces (such the Boss GP-10 for guitarsand the Zeta products for violins) typically only respond to thefundamental vibrating pitch and the overall amplitude. Further,amplitude responses in these current synthesizer interfaces typicallyonly respond to the amplitude at the initial attack of a note and theevent where the amplitude of the sustained vibration falls below acertain threshold.

[0658] The invention provides for an expansion of traditionalsynthesizer control interfaces for vibrating elements so as to respondto the pitches and amplitudes of higher-order overtone vibrations andissue control signals based on these. By expanding the response oftraditional synthesizer interfaces for vibrating elements to includecontinuous time response to fundamental and overtone amplitudes as wellas pitches, far more expressive control over synthesized sound viatracking of vibrating elements can be obtained. For example, plucking orbowing a string in varying locations can be used to control signalprocessing parameters.

[0659] Traditionally, synthesizer interfaces for vibrating elementscapture pitch (based on fundamental frequency of vibration) andamplitude, initially when a vibrating element is excited and in somecases as pitch and/or even amplitude changes dynamically. However, thiscan be expanded to include responses to various higher-order(non-fundamental) harmonics or other modes of vibration. It is importantto note that such a feature can add tremendous control over conventionalsynthesizer sound production in general situations where vibratingelements are used to control the synthesis of the sound; this is truesomewhat in guitars, but much more so in wind and bowed instruments. Inusing a vibrating element feedback excitation arrangement for guitars,for example, this type of control signal extraction may be especiallyexpressive as the feedback process can create widely-varying harmoniccontent when hands touch vibrating elements in feedback excitation or byvarying the excitation feedback characteristics (via signal processingwithin the feedback loop). Because of the dynamic overtonecharacteristics of exciting vibrating elements in feedback loops, it isof interest to expand traditional synthesizer interfaces for vibratingelements to respond to the pitches and amplitudes of these higher-orderovertone vibrations. The significant synergistic value of thecombination of vibrating element excitation and overtone trackingcontrol signal extraction are also recognized as part of the invention.

[0660] There are various ways to accomplish such overtone tracking. Ingeneral, it is much easier for instruments whose elements vibrate atfixed pitches with a known overtone series. In these instruments, theovertone frequencies of a given vibrating element are also known inadvance. FIG. 64 shows an example method for the generation of controlsignals from fundamental and overtone information in a signal from avibrating element of fixed known pitch. If the pitch is both fixed andknown, the signal from each vibrating element can be filtered by a suiteof band-pass filters 6402.1-6402.h, each separately tuned to the knownfrequency of an individual mode of vibration for that particularelement. The output of each filter can be fed to a dedicated amplitudefollower 6403.1-6403.h. Each amplitude follower output can be used tocreate a separate parameter which can be assigned to an outgoing controlsignal 6405 via parameter mapping operation 6404.

[0661] The invention provides for the combining and/or processing offundamental and overtone information in creating yet other derivedcontrol signals. FIG. 65 shows combining and/or processing offundamental and overtone information obtained from a vibrating elementsignal prior to parameter assignment to control signals. As shown inFIG. 65, the outputs of groups of amplitude detectors associated with agiven vibrating element can be combined and/or processed 6406 beforemapping to final parameters prior to control signal assignment. Forexample, different weighted sums can be used to control the amplitude ofa synthesized signal (say a uniform averaging, or sum-of-squaresaveraging) than would be used to control the cut-off frequency of asubtractive filter (here, weighting the higher modes of vibration morestrongly would make the synthesis mimic the vibrating element's harmonicbalance; weighting the lower more strongly would make the synthesiscomplement the vibrating element's harmonic balance, etc.).

[0662] In the case where the vibrating elements do not vibrate at afixed pitch but still obey a known overtone relationship, a slightlymore involved version of the same mechanism can also be used. Note thatsuch an implementation is hardly limited to feedback systems and couldbe used in general guitar and violin synthesizer interfaces for newdepths of performance control. In addition, because variation inovertone series dynamics is an essential factor in singing and inpercussion instruments, such a technology opens important new doors forsynthesizer overtone-nuance tracking for voice and percussioninstruments. In singing in particular, the relative amplitudes of thefirst three harmonics (largely the first two, actually) determine thechoice of sung vowel; as a result, this technology allows synthesizersto track the formants of vowel production in the human voice.

[0663]FIG. 66 shows an example implementation of an adaptive method fortracking overtones for a variable-pitch vibrating element with knownovertone series. The method is largely the same as the fixed-pitch case,but with some added steps. The additional steps employed are to firstuse a traditional pitch detector 6407 (as used with conventional MIDIguitar/violin/voice interfaces) to determine the fundamental pitch, thenuse this pitch information plus an overtone series model of thevibrating element 6400 to position the frequencies of the individualband-pass filters 6402.1-6402.h and amplitude followers 6403.1-6403.h.

[0664] In a preferred implementation of this approach, the detectedpitch information provided by the pitch detector 6407 is fed to amodel-based overtone series calculator 608. The model-based overtoneseries calculator 155 generates the control signals required toindividually center each of the plurality of band-pass filters6402.1-6402.h. The model-based overtone series calculator 6408 is alsoused to generate overtone frequency information for use in any combiningor processing of the extracted overtone amplitude information and in theparameter mapping 6404 to final output control signals.

[0665] 7.6.2 Pluck direction to MIDI

[0666] The invention provides for the extraction of plucking direction(as on an instrument string) of arbitrary vibrating element and creatinga control signal from it. Core technologies for detecting pluckdirection typically include separate analysis of the signals from a2-coil humbuck pickup and have been implemented in products by Biax andPassaic. Passaic also implemented a method for deriving a controlparameter from where a string was plucked between the bridge and theneck. The invention provides for these extraction functions to beincluded in the available control extraction capabilities.

[0667] 7.6.3 Video Motion and Feature Extraction

[0668] The invention provides for the extraction of parameters fromprovided video signals as described earlier and creating control signalsfrom them. Methods for implementing this have been described earlier,including simple timing tests and video frame grabs analyzed bydedicated systems or personal computer software. The invention alsoprovides for implementations using emerging motion tracking and imagedecomposition methodologies under development for widespread adoption indigital video compression standards such as MPEG-4 (see for example[Hara; Bormans].

[0669] 7.6.4 Control Signal Pattern Recognition

[0670] The invention provides for the recognition of control signalpatterns. Since the result is yet another control signal, this has beentreated earlier in the context of control signal processing.

[0671] 7.7 Dynamic Control Signal Synthesis

[0672] The invention provides for the synthesis of dynamic controlsignals such as low-frequency sweep oscillators, particularly thosewhose parameters may be controlled in real-time by other controlsignals. Since an envelope generator trigger is also a control signal,the generation of control signal envelopes and slews are also includedin this category and are provided for by the invention.

[0673] 7.7.1 MIDI-controlled low-frequency control oscillators ensembles

[0674] Low-frequency sweep oscillators, or LFOs, have roles throughoutthe invention and have been discussed earlier. In some types offunctions implemented by specific elements, such as chorus and flangers,the LFO may be hard associated with the element. The invention providesfor this as well as the remote positioning of the LFO function outsidethe element in the case where several elements may be coordinated withthe same LFO. In other types of functions, such as location modulation,it may be best to control existing elements such as mixers with controlsignals from external LFOs.

[0675] The invention provides for a plurality of control signal LFOs tobe available. The LFOs may be part of a comprehensive system or aseparate module which can be manufactured and sold for other uses; sucha product would be naturally served by at least MIDI output and input,but may also include at least one analog input and/or output. The LFOsprovided for by the invention include multiple phase output capabilitiesas well as selections of a variety of waveforms, frequency settings,amplitude settings and offset settings, all of which may be varied inreal-time by yet other control signals. Further, the invention providesfor these parameters to be available under selectable stored programcontrol which may be chosen by control signals. Finally, the inventionprovides for global effects across groups of LFOs, such as timing slewof parameter changes, global scaling, global offsets, etc. These mayalso include more complex organizations such as may be require fortwo-dimensional and three-dimensional location modulation and the customconstruction or sampling of LFO waveforms.

[0676]FIG. 67 illustrates an example approach wherein a plurality ofLFOs with features as prescribed by the invention may be implemented.Program data for M different stored programs may be stored in M datastructures 6700.1- 6700.M; these may include specific LFO parameters6701.1.1-6701.M.N for N LFOs. The data structures may also includeglobal information 6702.1-6702.M pertaining to groups of LFOs. Areceived control signal and/or panel control may be used to designatethe selected program 6703 which is recalled and implemented 6704 as aset of default vaules subject to real-time change by other controlsignals. The LFO data vector structure for each LFO includes a source ofselect information 6711 for choosing whether a given output operates asan independent LFO or as a slave to another LFO specified here (and inso doing becomes a multiple phase output for the chosen sourceoscillator). Another part 6712 of the LFO data vector provides afrequency setting if the LFO is independent and a phase setting ifslaved; should integer-ratio phase-locking be implemented, this part ofthe data structure may be reorganized to include relative frequency andphase settings with respect to the selected master LFO. Another part6713 of the LFO data vector provides selected waveform information,including reference to any user sampled LFO waveforms. Another two parts6714, 6715 of the LFO data vector provide respectively amplitude scalingand offset settings. Additional information, such as the outgoing MIDIchannel and MIDI Continuous Controller number to be used and whatincoming MIDI Continuous Controller messages on what MIDI channel areused to control the aforementioned LFO settings. The global part of thedata 6731, 6732 provides global information for specific settings ofglobal amplitude, offset, parameter, time slew, etc. pertaining tospecific groups of LFOs. The selected information is presented to theLFO engine for execution, The LFO engine in the example implements anindependent LFO by dividing 6751 a system clock signal 6741 by a numberdetermined by the frequency setting 6712.

[0677] The divided clock signal runs a counter 6752 (here a 128 stepcounter is illustrated, although higher resolution may be desired).Should the LFO be instead designated, via the information 6711, to beslave to another LFO, the counter of that master LFO is accessed 6753and the phase offset information 6712 is used to create a phase offsetvalue which is provided together with the accessed counter value to anmodular adder 6755 which produces the resulting phase-shifted version ofthe master LFO count value. The resulting counting sequence produced bythis section 6750 is used as the address for a waveform look-up table6756 and/or algorithm; the resulting generated periodic signal is thenscaled 6757 according to 6714 and offset in amplitude 6758 byinformation 6715. The resulting waveforms can be post-processed toprovide global amplitude and offset features, or, alternative,mathematical transformations 6731 may be provided on the information6714, 6715 before executing it in the LFO engine in elements 6757, 6758.

[0678] 7.7.2 Controlled slews, ramp, and envelope generator elements

[0679] The invention provides for slew limiters, ramp generators, andenvelope generators whose trigger and various parameters may be variedin real-time by control signals. Slew limiters limit the rate of changeof a control signal to a maximal range which may be set as a parameterand advantageously varied by control signals. Ramp generators aresimplified envelope generators triggered by control signals which rampbetween two or more discrete values or the entire control signal rangeand do so according selected types of dynamics (linear over time,exponential over time, etc.); the parameters here may be set andadvantageously varied by control signals. Envelope generators offer morecomplex transient waveforms, typically including at least attack, decay,sustain, and release; more complex envelope features including morebreakpoints, delays, and segment curve shapes may also be provided. Theparameters here may be set and advantageously varied by control signals.

[0680] 7.8 Lighting Effects and Video Display

[0681] The invention provides for extensive control of lighting viacontrol signals. Some aspects of lighting as provided for by theinvention are described in the sub-sections below.

[0682] 7.8.1 Light Types

[0683] The invention provides for extensive control of lighting viacontrol signals. Some aspects of lighting as provided for by theinvention are described in the sub-sections below.

[0684] 7.8.1.1 Traditional fixed

[0685] The invention provides for traditional fixed lightingarrangements as shown in FIG. 68.

[0686] These may include any one or more of overhead lights 6802,far-throw lights 6807, foot lights 6803, floor lights aimed upward or atangles 6804, backlights 6806 behind equipment and risers 6805 andbackdrops 6801.

[0687] 7.8.1.2 Movable

[0688] The invention provides for movable lighting controlled inreal-time via control signals.

[0689] Such lighting can be implemented by attaching lights to motorizedpan/tilt heads as used for video cameras.

[0690] 7.8.1.3 Instrument lighting

[0691] The invention provides for lighting on instrument entities whichmay be operated via control signals. FIG. 69 shows examples of lightingfor a guitar 6900 from the bridge 6901, neck 6902, above 6904 and below6903 the picking area, all aimed at illumination effects for the hands.Also shown are lights aimed at the audience in the pickup areas 6905,6906 and fret areas 6907, any of which may be aggregate as in 6905, 6906or split out separately for each string as in 6907.

[0692] 7.8.1.4 Light sculptures

[0693] The invention provides for light sculptures under control ofcontrol signals. FIG. 70 shows rotating speaker emulation lightsculptures. In one implementation a rotating reflective beacon 7011reflects gathered light from a bulb 7012 and projects it on to atranslucent concealing cover 7013 attached to, in this case, a pyramidframe. An alternate arrangement where the translucent outer structure7023 itself rotates is also shown about a fixed light bulb 7022 whosecable 7026 fits through the bearings 7024, 7025 of a rotating turntable7021, 7024 driven by geared 7027, 7028 motor arrangement. The mechanismjust described is exploded for explanation and in fact may be readilycollapsed by standard means so as to permit two rotating pyramids 70317032 to be stacked in transparent cubes 7033, 7034, the motor speed anddirection can be controlled by control signals and arranged to operatein synchronization with rotating speaker simulations in audio signalprocessing elements/FIG. 71 shows light pyramids 7100 made of similarelements 7101-7106, typically without reflectors or directional bulbs,which may be arranged in arrays 7108. Also shown are light column arrayssuggestive of organ pipes 7118 or instrument strings 7119. These lightcolumns may be built with standard lamps and reflectors 7112, 7114,7115, colored gels 7111 and light transmitting and scattering rods 7110.

[0694] Also provided for by the invention are controlled ionize gasturbulence sculptures; these may be used with or without associatedvideo ameras.

[0695] 7.8.2 General Lighting Control

[0696] The invention provides for lights to be used in scene changemodes or modulated by control signals according to:

[0697] animation sequences and subsequence events

[0698] instrument activity

[0699] timbre qualities

[0700] Special Instrument lighting effects include:

[0701] audience shock events

[0702] animation sequences

[0703] string activity, note following, orchestration following

[0704] 7.8.3 Video Signal Routing

[0705] Referring to FIGS. 1-2, input signals directed to video routinginclude the video outputs from the instrument entities 100, video signalprocessing elements 127, video signal synthesis elements 129 a, andexternal video feeds from, for example, miscellaneous stage cameras,VCRs, etc. Still referring to FIGS. 1-2, output signals directed fromvideo routing include video inputs to instrument entities 100, videosignal processing 127, control signal extraction 128 a, and overallvideo outputs to one or more of any displays, projectors, and/orrecording facilities.

[0706] 7.8.4 Video Signal Processing

[0707] Video signal processing as provided for by the invention wouldinclude overlays, wipes, fades, blends, solarizations, geometricwarping, etc, as much as possible under the control of control signals.Interesting effects provided for by the invention include the switching,wiping, blending, fading, warping, etc. of various video signals fordisplay in performance and/or recording under the control of instrumentnote and amplitude envelope signals.

[0708] 7.8.5 Video Display

[0709]FIG. 72 shows how the invention provides for video projection tobe used to shine down 7201 on the stage 7200, shine horizontally 7202 onto performers or backdrops. The invention also provides for videoprojectors to be shined on the audience. The invention also provides forThe invention provides for movable cameras controlled in real-time viacontrol signals using motorized pan/tilt heads as well as motorizedzoom/focus lenses.)

[0710] 7.8.6 Video Signal Synthesis

[0711] The invention provides for video signal synthesis would includereal-time generation of text message screens, text overlays, vector andraster graphic drawings, vector and raster graphic overlays, andanimations affiliated with numerical dynamics simulation. The inventionalso provides for pre-stored video frames, playback of video clips, andplayback of prestored vector and raster graphics animations. Theinvention provides for these to be controlled by standardized controlsignals, such as MIDI, and as such would typically involve both storedprogram control and parameterized control. These functions may berealized with a conventional personal computer fitted with video cardand MIDI interface as well as by dedicated hardware.

[0712] 8 Example Envisioned Applications

[0713] A few example envisioned applications of the invention are nowprovided.

[0714] 8.1 Add-on Modules for existing instruments

[0715] This gives rise to a whole new marketplace for new instruments,instrument retrofit kits, and music signal processor units which caninteract with external amplifiers, signal processing, and MIDIsynthesizer units.

[0716] 8.2 Creation of enhanced electronic vibrating element instruments

[0717] With the first technique described within this patent, the moretraditional acoustically-excited “controlled feedback” effects caneasily be obtained, via electromagnetic excitation, with standard parts.Specialization of the parts can provide additional features. Thetechnique can also be applied to any instrument where sound is produceby vibrating ferromagnetic material, e.g., African mbiras, violins,xylophones, etc.

[0718] With the second technique described in this patent, conventionalsignal processing can be used on each string signal to create“generalized pedal steel guitars,” multi-modal Indian sitars (wheredrone and sympathetic strings can be electronically retuned whileplaying), spatially animated string sounds within a stereophonic orspatial sound field, and mixed timbre instruments where different signalprocessing methods are applied to each string.

[0719] The technique can also be applied to any instrument wherevibration of individual sound-producing elements can be electronicallycaptured by isolated transducers (electromagnetic, optical, Hall-effect,etc.), such as nylon-stringed instruments, marimbas, African mbiras,violins, etc.

[0720] By combining these two new techniques with appropriate signalprocessing, a very powerful environment for multi-stringed electronicinstruments can be created. Individual strings can be singled out forfeedback operation while others operate without feedback, and allstrings can be electronically pitch-shifted as needed in a performance.The results allow a performer a greater degree of polyphonic control,using mechanical (neck, frets, fingers, picks, movable tailpieces, pedaltuning changers, etc.) or electronic means for both string excitationand pitch control, with individual string outputs available forsynthesizer interfaces.

[0721] Any to all of the above can be built into an individualinstrument. Alternatively, an instrument interface can be created andmost signal processing can be remotely located from the instrument,connecting to it via this interface. If this interface is standardizedacross multiple instruments, then common signal processing equipmentenvironment can be used across a wide variety of instruments(metal-stringed and nylon-stringed guitars, basses, violins, steelguitars, sitars, mbiras, etc.). This gives rise to a whole newmarketplace for new instruments, instrument retrofit kits, and musicsignal processor units which can interact with external amplifiers,signal processing, and MIDI synthesizer units.

[0722] All publications and patent applications mentioned in thisspecification are herein incorporated by reference to the same extent asif each individual publication or patent application was specificallyand individually indicated to be incorporated by reference. Theinvention now being fully described, it will be apparent to one ofordinary skill in the art that many changes and modifications can bemade thereto without departing from its spirit or scope.

REFERENCES CITED

[0723] The following references are cited in this patent applicationusing the format of the first one or two authors last name(s) withinsquare brackets “[ ]”, multiple references within a pair of squarebrackets separated by semicolons “;”

[0724] [Bicanic] Bicanic, Dane, ed. Photoacoustic and PhotothermalPhenomena III, Vol. 69, Springer-Verlag, Berlin Heidelberg, 1992.

[0725] [Ronse] Ronse, Christian and Devijver, Pierre A., ConnectedComponents in Binary Images: the Detection Problem, John Wiley & SonsInc. New York, 1984.

[0726] [Viberg] Viberg, Mats, Subspace Fitting Concepts in Sensor ArrayProcessing, Linkoping Studies in Science and Technology. DissertationsNo. 27 Linkoping, Sweden 1989.

[0727] [Shapiro] Shapiro, Larry S, Affine Analysis of Image Sequences,Cambridge University Press, 1995.

[0728] [Appelman] Appelman, D. Ralph, The Science of Vocal Pedagogy,Theory and Applications, Indiana University Press, Bloomington, 1967.

[0729] [Murphy] Murphy, Aamodt, Spicer and Royce (eds.) Photoacousticand Photothermal Phenomena II, Vol. 62, Springer-Verlag, BerlinHeidelberg, 1990.

[0730] [Winckel] Winckel, Fritz, Music Sound, and Sensation: A ModernExposition, Dover, N.Y., 1967.

[0731] [Hara] Hara, Yoshiko “Matsushita demos multilayer MPEG-4compression”, Electronic Engineering Times, Apr. 19, 1999.

[0732] [Lusher] Lusher, E. (unk) “Photoacoustic effect: principles andapplications: proceedings of the First International Conference on thePhotoacoustics,” Vieweg, 1984.

[0733] [Wang] Wang, X-J, and Nicolis, G. “Bifurcation Phenomena inCoupled Chemical Oscillators: Normal Form Analysis and NumericalSimulations”, Physica 26D (1987) 140-155 North-Holland, Amsterdam.

[0734] [Kuramoto] Kuramoto, Yoshiki, “Rhythms and Turbulence inPopulations of Chemical Oscillators”, Physica 106A (1981) 128-143North-Holland.

[0735] [Epstein] Epstein, Irving R, “New Chemical Oscillators”,Non-Equilibrium Dynamics in Chemical Systems, Springer-Verlag,Berlin/Heidelberg 1984.

[0736] [Gray] Gray, Peter, and Scott, Stephen K, Chemical Oscillationsand Instabilities, Clarendon Press, Oxford 1990.

[0737] [Epstein] Epstein, “The Search for New Chemical Oscillators,” inChemical Instabilities, Nicolis and Baras (eds.), D.Reidel Publishing,Dordrecht/Boston/Lancaster, 1984.

[0738] [Nicolis] Nicolis, G and Baras, F (eds.), Chemical Instabilities,D.Reidel Publishing, Dordrecht/Boston/Lancaster.

[0739] [Tyson] Tyson, John J., The Belousov-Zhabotinskii Reaction,Springer-Verlag, Berlin/Heidelberg.

[0740] [Nicolis] Nicolis, G., “Nonlinear Phenomena in Chemistry”,Introduction to Nonlinear Science, Cambridge University Press, GreatBritain 1995.

[0741] [Orban] Orban, M., Kurin-Csorgei, K., Zhabotinsky, A, andEpstein, I, “New Indicators for Visualizing Pattern Formation inUncatalyzed Bromate Oscillatory Systems”, J. Am. Chem. Soc. 1998, 120,1146-1150.

[0742] [Field] Field, Richard J., and Noyes, Richard M “Oscillations inChemical Systems IV Limit Cycle Behavior in a Model of a Real ChemicalReaction” J. Chem.. Physics Vol. 60 No. 5 Mar. 1974.

[0743] [Nagy-Ungvarai] Nagy-Ungvarai, Z. and Hess, Benno “Control ofDynamic Pattern Formation in the Belousov-Zhabotinsky Reaction” PhysicaD (1991) North-Holland.

[0744] [Ruoff] Ruoff, P. “How Bromate Oscillators Are Controlled”, Am.Chem. Society 1988.

[0745] [Bormans] Bormans, Jan “MPEG-4 systems need specialized CPUs”,Electronic Engineering Times, Jan. 25, 1999.

[0746] [Visintin] Visintin, A (ed.), Models of Hysteresis, LongmanScientific & Technical, Harlow, England, 1993.

1. A system for the generation of control signals used to control audio,visual and control systems based on the processing of image and sensorarray data, the system comprising: a source of at least one data arraycomprising pixel data comprising scalar-valued or vector-valued data; atleast one processor for processing said data array; an algorithm and/orcircuitry for extracting information comprising at least one of patternboundaries; pattern geometric coordinates; geometric weighted momentsdefined by the pattern and the pixel data in combination; patternclassifications, wherein the extracted information is used to createcontrol signals.
 2. A method for the generation of control signals usedto control audio, visual and control systems based on the processing ofimage and sensor array data, the system utilizing: a source of at leastone data array comprising pixel data utilizing scalar-valued orvector-valued data; at least one processor for processing said dataarray; an algorithm and/or circuitry utilized to extract informationcomprising at least one of: pattern boundaries; pattern geometriccoordinates; geometric weighted moments defined by the pattern and thepixel data in combination; pattern classifications, wherein theextracted information is used to create control signals.
 3. The systemof claim 1 wherein the source of said data array is a pressure sensorarray.
 4. The system of claim 3 wherein said pressure sensor array isattached to the key of a musical instument keyboard.
 5. The system ofclaim 1 wherein the source of said data array is a video camera.
 6. Thesystem of claim 1 wherein the source of said data array is any one of: achemical sensor array; an enviromental sensor array; or a body sensorarray.
 7. The system of claim 1 wherein the process for generation ofcontrol signals from the extracted information is variably defined. 8.The system of claim 7 wherein a plurality of variably defined generationprocesses may be stored and recalled.
 9. The method of claim 2 whereinthe process for generation of control signals from the extractedinformation is variably defined.
 10. The method of claim 10 wherein aplurality of variably defined generation processes may be stored andrecalled.